Nov 03 18:41:46.427856 ip-10-0-137-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 18:41:46.427866 ip-10-0-137-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 18:41:46.427873 ip-10-0-137-192 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 18:41:46.428099 ip-10-0-137-192 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 18:41:56.444490 ip-10-0-137-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 18:41:56.444508 ip-10-0-137-192 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 576bcac4ade840a68302e8a4142c8c94 --
Nov 03 18:44:17.312316 ip-10-0-137-192 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 18:44:17.863581 ip-10-0-137-192 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:44:17.863581 ip-10-0-137-192 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 18:44:17.863581 ip-10-0-137-192 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:44:17.863581 ip-10-0-137-192 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:44:17.867889 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867808    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.867889 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867889    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867900    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867905    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867909    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867913    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867916    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867919    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867922    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867925    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867928    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867931    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867933    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867938    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867942    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867945    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867951    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867953    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867956    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.867961 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867959    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867962    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867965    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867968    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867971    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867974    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867978    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867980    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867983    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867987    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867989    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867992    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867995    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.867997    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868000    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868003    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868006    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868008    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868011    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868013    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.868407 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868028    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868031    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868034    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868036    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868039    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868042    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868045    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868048    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868051    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868053    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868056    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868059    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868061    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868065    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868068    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868071    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868074    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868077    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868079    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.868901 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868082    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868085    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868087    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868090    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868093    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868095    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868098    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868101    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868103    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868106    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868109    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868111    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868114    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868117    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868120    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868123    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868125    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868128    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868131    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.869375 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868135    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868139    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868142    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868145    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868148    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868153    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868156    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868160    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868162    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868165    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868168    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868171    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868173    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868176    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868179    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868182    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868185    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868187    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868190    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.869816 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868193    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868196    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868199    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868202    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868204    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868209    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868212    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868215    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868985    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868991    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868995    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.868998    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869001    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869004    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869007    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869010    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869013    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869029    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869032    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869035    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.870305 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869038    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869041    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869044    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869047    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869050    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869052    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869055    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869058    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869061    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869063    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869066    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869069    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869072    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869075    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869078    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869082    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869085    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869087    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869090    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.870777 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869093    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869095    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869098    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869101    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869104    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869108    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869112    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869116    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869120    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869123    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869126    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869129    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869131    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869134    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869137    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869141    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869143    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869146    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869149    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.871225 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869152    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869154    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869157    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869159    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869162    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869165    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869167    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869170    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869173    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869176    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869179    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869182    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869184    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869188    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869190    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869193    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869196    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869199    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869201    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869204    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.871667 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869207    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869210    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869213    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869215    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869218    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869221    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869224    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869226    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869229    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869232    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869234    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869237    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869241    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869244    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869248    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869251    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869254    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869257    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869260    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.872154 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869263    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869266    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869269    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869271    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869274    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869277    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869280    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869282    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869285    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869288    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869290    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869293    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869297    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869299    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869302    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869305    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869308    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869819    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869825    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869829    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.873443 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869833    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869836    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869838    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869842    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869844    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869848    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869850    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869853    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869855    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869859    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869865    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869867    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869870    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869873    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869876    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869878    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869881    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869884    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.873922 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869887    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869889    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869892    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869895    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869901    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869905    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869908    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869912    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869915    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869918    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869921    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869924    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869927    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869930    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869932    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869935    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869937    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869940    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.874489 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869943    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869945    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869948    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869951    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869954    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869959    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869962    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869964    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869967    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869971    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869973    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869976    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869979    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869982    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869985    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869987    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869990    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869993    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869996    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.869998    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.874983 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870001    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870004    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870009    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870012    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870030    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870034    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870037    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870041    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870044    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870047    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870050    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870053    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870055    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870058    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870061    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870064    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870066    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870070    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870073    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.875470 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870076    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870079    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870082    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870084    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870087    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870090    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870092    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870095    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870098    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870101    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870104    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870106    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870109    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870112    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870114    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870117    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870121    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870124    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870127    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870130    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.875952 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870132    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870135    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870138    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870140    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870143    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.870145    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871288    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871299    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871308    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871313    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871321    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871324    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871329    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871334    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871337    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871340    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871344    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871347    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871351    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871355    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871358    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871362    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871365    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871368    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 18:44:17.876434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871372    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871375    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871379    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871382    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871385    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871391    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871394    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871397    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871401    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871404    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871407    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871410    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871414    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871417    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871421    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871424    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871427    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871430    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871435    2579 flags.go:64] FLAG: --enable-server="true"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871438    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871443    2579 flags.go:64] FLAG: --event-burst="100"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871447    2579 flags.go:64] FLAG: --event-qps="50"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871450    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871453    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 18:44:17.876968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871456    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871460    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871463    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871468    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871471    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871474    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871478    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871481    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871484    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871487    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871490    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871493    2579 flags.go:64] FLAG: --feature-gates=""
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871498    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871501    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871505    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871508    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871512    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871515    2579 flags.go:64] FLAG: --help="false"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871518    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871522    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871525    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871528    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871531    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 18:44:17.877590 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871535    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871538    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871541    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871546    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871549    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871552    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871555    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871558    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871561    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871564    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871567    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871570    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871575    2579 flags.go:64] FLAG: --lock-file=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871578    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871581    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871585    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871590    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871593    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871597    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871600    2579 flags.go:64] FLAG: --logging-format="text"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871603    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871607    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871610    2579 flags.go:64] FLAG: --manifest-url=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871613    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871618    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 18:44:17.878181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871621    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871625    2579 flags.go:64] FLAG: --max-pods="110"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871628    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871631    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871635    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871638    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871641    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871644    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871647    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871655    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871659    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871662    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871666    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871669    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871672    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871675    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871678    2579 flags.go:64] FLAG: --port="10250"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871681    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871685    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e84d7b1a247a8c13"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871689    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871692    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871695    2579 flags.go:64] FLAG: --register-node="true"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871698    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871702    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871705    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871708    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 18:44:17.878781 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871711    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871715    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871719    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871722    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871725    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871728    2579 flags.go:64] FLAG: --runonce="false"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871732    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871735    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871738    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871741    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871744    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871747    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871751    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871754    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871757    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871760    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871764    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871767    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871770    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871773    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871776    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871781    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871784    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871787    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871791    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 03 18:44:17.879516 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871796    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871799    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871802    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871805    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871808    2579 flags.go:64] FLAG: --v="2"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871812    2579 flags.go:64] FLAG: --version="false"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871816    2579 flags.go:64] FLAG: --vmodule=""
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871820    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.871823    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871966    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871970    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871973    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871977    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871981    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871986    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871989    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871992    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871994    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.871997    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872000    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872002    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.880127 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872005    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872008    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872010    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872013    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872027    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872030    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872032    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872035    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872038    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872040    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872043    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872048    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872050    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872053    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872056    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872058    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872062    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872064    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872067    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872070    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.880617 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872073    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872076    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872078    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872081    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872084    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872086    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872089    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872092    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872095    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872098    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872100    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872103    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872106    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872109    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872111    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872114    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872117    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872119    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872122    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.881087 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872125    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872127    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872130    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872133    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872137    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872140    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872142    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872145    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872148    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872150    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872153    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872156    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872159    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872162    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872164    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872167    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872170    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872173    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872175    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.881536 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872178    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872181    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872184    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872186    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872189    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872192    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872195    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872197    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872200    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872202    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872205    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872208    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872210    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872213    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872216    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872219    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872221    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872225    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872227    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.881986 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872230    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872233    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872236    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872238    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872242    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872245    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872248    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872251    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872253    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872256    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872259    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872262    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872264    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872267    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872270    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872273    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.882449 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872277    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.872306    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872343    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872348    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872351    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872357    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872360    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872363    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872366    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872369    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872372    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872374    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872378    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872383    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872386    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.882821 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872389    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872392    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872396    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872399    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872402    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872406    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872409    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872412    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872414    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872417    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872420    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872423    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872425    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872428    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872431    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872434    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872437    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872440    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872443    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.883201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872446    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872448    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872451    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872454    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872457    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872459    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872462    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872465    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872468    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872471    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872473    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872476    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872479    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872482    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872485    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872488    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872491    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872493    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872498    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872501    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.883668 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872503    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872506    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872509    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872511    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872514    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872517    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872519    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872522    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872525    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872528    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872531    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872534    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872537    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872540    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872543    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872546    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872548    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872551    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872553    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.884165 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872556    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872559    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872561    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872564    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872567    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872570    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872573    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872576    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872578    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872581    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872584    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872587    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872590    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872593    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872596    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872599    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872602    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872604    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872607    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.884599 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872610    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872612    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872615    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872618    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872620    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872624    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872627    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872629    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872632    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872635    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872637    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872640    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872643    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872645    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.872648    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.879436    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.879450    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.879466    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.879470    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 18:44:17.885085 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879529    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879535    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879539    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879542    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879545    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879548    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879551    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879554    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879556    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879559    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879562    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879566    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879570    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879574    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879576    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879579    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879582    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879585    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879588    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.885532 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879591    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879594    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879596    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879599    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879602    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879605    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879608    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879610    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879613    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879616    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879619    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879622    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879625    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879628    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879631    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879633    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879636    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879639    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879642    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879645    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.885965 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879647    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879650    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879653    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879655    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879658    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879661    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879663    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879668    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879671    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879675    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879678    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879681    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879685    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879687    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879690    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879693    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879695    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879698    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879701    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.886435 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879704    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879707    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879710    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879713    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879716    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879718    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879721    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879724    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879727    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879729    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879732    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879735    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879738    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879740    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879743    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879746    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879748    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879751    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879754    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879757    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.886879 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879759    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879762    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879765    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879767    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879770    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879773    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879776    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879779    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879781    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879784    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879787    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879789    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879792    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879795    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879798    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879801    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879804    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879807    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879809    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879812    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.887350 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879815    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879817    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879820    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879823    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879825    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879828    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879831    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.879833    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.879862    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880038    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880044    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880048    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880050    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880054    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880057    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.887833 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880060    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880062    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880065    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880068    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880071    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880074    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880077    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880080    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880083    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880085    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880088    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880092    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880095    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880097    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880100    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880103    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880106    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880109    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880112    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880115    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.888201 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880118    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880123    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880126    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880128    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880131    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880134    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880137    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880140    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880143    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880145    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880148    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880151    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880154    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880156    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880159    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880162    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880164    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880168    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880170    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880173    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.888712 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880176    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880178    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880181    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880184    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880187    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880190    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880193    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880195    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880198    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880201    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880204    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880206    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880209    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880211    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880214    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880216    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880219    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880222    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880224    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880227    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.889555 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880230    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880232    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880235    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880238    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880240    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880243    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880246    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880249    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880251    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880254    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880257    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880260    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880262    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880265    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880268    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880271    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880274    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880276    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880279    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.890279 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880281    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880284    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880287    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880289    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880292    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880295    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880297    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880300    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880303    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880306    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880308    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880311    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880314    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880316    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880319    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880322    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880325    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880327    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880330    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.890719 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880335    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880338    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.880363    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880400    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880405    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880409    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880412    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880414    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880417    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880420    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880423    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880426    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880429    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880431    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880434    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880437    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:44:17.891184 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880439    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880442    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880446    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880450    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880453    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880456    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880458    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880461    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880464    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880467    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880469    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880472    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880475    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880477    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880480    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880482    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880486    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880488    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880491    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:44:17.891563 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880494    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880497    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880499    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880502    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880505    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880508    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880511    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880514    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880516    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880519    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880522    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880524    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880527    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880530    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880532    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880535    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880538    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880540    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880543    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880546    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:44:17.892191 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880548    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880551    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880553    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880556    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880559    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880561    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880564    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880567    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880569    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880573    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880576    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880578    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880581    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880583    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880586    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880589    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880592    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880597    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880600    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:44:17.892836 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880602    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880605    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880608    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880610    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880613    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880616    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880619    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880622    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880624    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880627    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880630    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880633    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880636    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880639    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880641    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880644    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880647    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880649    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880652    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880655    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:44:17.893406 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880657    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880660    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880664    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880668    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880671    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880673    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880676    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880679    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880681    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880684    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880688    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880691    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880694    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880698    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:17.880701    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.880751    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.883273    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.884615    2579 server.go:1014] "Starting client certificate rotation"
Nov 03 18:44:17.893903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.884705    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:44:17.894408 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.885907    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:44:17.894408 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.887450    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:44:17.894408 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.893287    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:44:17.908982 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.908961    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 03 18:44:17.912713 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.912697    2579 log.go:25] "Validated CRI v1 image API"
Nov 03 18:44:17.913507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.913479    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 18:44:17.916586 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.916568    2579 fs.go:135] Filesystem UUIDs: map[2004e218-964e-4db5-b328-8251b3504349:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8a1b3ee9-8eea-4034-98d9-60c9c51d0892:/dev/nvme0n1p4]
Nov 03 18:44:17.916640 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.916586    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 18:44:17.923155 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.923047    2579 manager.go:217] Machine: {Timestamp:2026-11-03 18:44:17.920206262 +0000 UTC m=+0.448363364 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099822 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f08ce56531714359b90e317507f5f SystemUUID:ec2f08ce-5653-1714-359b-90e317507f5f BootID:576bcac4-ade8-40a6-8302-e8a4142c8c94 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:a8:9b:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:a8:9b:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:df:56:10:29:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 18:44:17.923155 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.923154    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 18:44:17.923287 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.923272    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 18:44:17.926728 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926709    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 18:44:17.926859 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926730    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-192.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 18:44:17.926903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926862    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 18:44:17.926903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926870    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 18:44:17.926903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926886    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:44:17.926903 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.926901    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:44:17.928085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.928075    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 18:44:17.931487 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.931476    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 18:44:17.931609 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.931600    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 18:44:17.933548 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933538    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 18:44:17.933597 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933552    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 18:44:17.933597 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933566    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 18:44:17.933597 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933576    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 03 18:44:17.933597 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933584    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 18:44:17.933939 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.933924    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:44:17.936477 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.936461    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 18:44:17.945895 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.945877    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:44:17.947278 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947266    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 18:44:17.947324 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947297    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 18:44:17.947442 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947432    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 18:44:17.947473 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947448    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 18:44:17.947473 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947455    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 18:44:17.947473 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947461    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 18:44:17.947473 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947467    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 18:44:17.947473 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.947473    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 18:44:17.949936 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 18:44:17.949973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949937    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 18:44:17.949973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949945    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 18:44:17.949973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949951    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 18:44:17.949973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949961    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 18:44:17.949973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.949970    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 18:44:17.951222 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.951207    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 18:44:17.951254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.951234    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 18:44:17.955244 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.955233    2579 server.go:1263] "Started kubelet"
Nov 03 18:44:17.955366 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.955295    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 18:44:17.955659 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.955493    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 18:44:17.955659 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.955577    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 18:44:17.956112 ip-10-0-137-192 systemd[1]: Started Kubernetes Kubelet.
Nov 03 18:44:17.956658 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.956563    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 18:44:17.957997 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.957980    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 18:44:17.961891 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:17.961866    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 18:44:17.962751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.962736    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 18:44:17.963904 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.963881    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 18:44:17.964028 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:17.963995    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:17.964628 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964604    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 18:44:17.964628 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964605    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 18:44:17.964751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964633    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 18:44:17.964751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964697    2579 factory.go:55] Registering systemd factory
Nov 03 18:44:17.964751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964708    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 18:44:17.964751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964718    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 18:44:17.964751 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964711    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 03 18:44:17.964944 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964933    2579 factory.go:153] Registering CRI-O factory
Nov 03 18:44:17.964944 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964944    2579 factory.go:223] Registration of the crio container factory successfully
Nov 03 18:44:17.965001 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964985    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 18:44:17.965001 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.964999    2579 factory.go:103] Registering Raw factory
Nov 03 18:44:17.965092 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.965011    2579 manager.go:1199] Started watching for new ooms in manager
Nov 03 18:44:17.965908 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.965891    2579 manager.go:319] Starting recovery of all containers
Nov 03 18:44:17.977777 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.977759    2579 manager.go:324] Recovery completed
Nov 03 18:44:17.979339 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:17.979314    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 18:44:17.982358 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.982343    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:17.984155 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984140    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:17.984222 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984169    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:17.984222 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984183    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:17.984656 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984640    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 03 18:44:17.984656 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984646    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:17.984656 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984651    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 18:44:17.984789 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.984667    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 18:44:17.985363 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985346    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:17.985444 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985372    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:17.985444 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985382    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:17.985914 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985903    2579 policy_none.go:50] "Start"
Nov 03 18:44:17.985958 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985917    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 18:44:17.985958 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.985930    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 18:44:17.987212 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:17.987202    2579 policy_none.go:44] "Start"
Nov 03 18:44:18.019717 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.019578    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 18:44:18.023749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.023736    2579 manager.go:341] "Starting Device Plugin manager"
Nov 03 18:44:18.023825 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.023762    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 18:44:18.023825 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.023772    2579 server.go:87] "Starting device plugin registration server"
Nov 03 18:44:18.023976 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.023966    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 18:44:18.024006 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.023980    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 18:44:18.024096 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.024077    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 18:44:18.024198 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.024161    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 18:44:18.024198 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.024169    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 18:44:18.024669 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.024649    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 18:44:18.024751 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.024691    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:18.094611 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.094548    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 18:44:18.095968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.095762    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 18:44:18.096569 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.096548    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 18:44:18.096634 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.096589    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 18:44:18.096675 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.096629    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 18:44:18.124572 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.124551    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:18.125332 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.125317    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:18.125404 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.125346    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:18.125404 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.125357    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:18.125404 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.125381    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:18.155444 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.155416    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-192.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 18:44:18.156192 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.155103    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-192.us-west-2.compute.internal.18e49d41e5dd125a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:17.955205722 +0000 UTC m=+0.483362821,LastTimestamp:2026-11-03 18:44:17.955205722 +0000 UTC m=+0.483362821,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-192.us-west-2.compute.internal,}"
Nov 03 18:44:18.196861 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.196833    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"]
Nov 03 18:44:18.196953 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.196888    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:18.197736 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.197721    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:18.197809 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.197756    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:18.197809 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.197766    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:18.199640 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.199626    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:18.199773 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.199759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:18.199841 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.199790    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:18.200317 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200297    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:18.200317 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200307    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:18.200428 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200329    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:18.200428 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200330    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:18.200428 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200339    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:18.200428 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.200355    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:18.201657 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.201642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:18.201746 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.201673    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:18.202378 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.202360    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:18.202453 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.202395    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:18.202453 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.202412    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:18.206134 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.205974    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-192.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 18:44:18.224599 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.224577    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 18:44:18.224687 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:18.224631    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzhcd"
Nov 03 18:44:18.235313 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.235270    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 18:44:18.235407 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.235367    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-192.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 18:44:18.235623 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.235550    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-192.us-west-2.compute.internal.18e49d41e796ca68  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-192.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:17.984154216 +0000 UTC m=+0.512311316,LastTimestamp:2026-11-03 18:44:17.984154216 +0000 UTC m=+0.512311316,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-192.us-west-2.compute.internal,}"
Nov 03 18:44:18.237152 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.237135    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:18.243930 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:18.243910    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.723280 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.723260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.723372 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.723286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.723372 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.723313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.737954 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:19.737932    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 03 18:44:19.768419 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.768396    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:44:19.769013 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:19.768996    2579 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-192.us-west-2.compute.internal\" not found" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824369 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824369 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824369 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824355    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824511 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824511 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.824511 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.824463    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.912736 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:19.912710    2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-192.us-west-2.compute.internal\" not found" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.969511 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.969487    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:19.970482 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.970456    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:19.970587 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.970493    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:19.970587 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.970504    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:19.970587 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.970533    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:19.999908 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:19.999876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:20.005271 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.005050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:20.026556 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.026533    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:20.026556 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.026557    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-192.us-west-2.compute.internal\": node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.136952 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.136852    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.237648 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.237614    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.338013 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.337988    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.341144 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.341127    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 18:44:20.341252 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.341237    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 18:44:20.341390 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.341308    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a3904a6b916d941eabd9d7e63c223c68-8c1f0edaac096962.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-192.us-west-2.compute.internal.18e49d41e796ca68\": read tcp 10.0.137.192:52970->3.14.110.91:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-192.us-west-2.compute.internal.18e49d41e796ca68  default   5625 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-192.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-192.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-192.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 18:44:17 +0000 UTC,LastTimestamp:2026-11-03 18:44:18.19773611 +0000 UTC m=+0.725893209,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-192.us-west-2.compute.internal,}"
Nov 03 18:44:20.438970 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.438913    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.539632 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.539582    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.576622 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.576600    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:44:20.579354 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.579337    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:44:20.579422 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.579368    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:44:20.579422 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.579378    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:44:20.639848 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.639822    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.686252 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.686222    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:39:19 +0000 UTC" deadline="2028-07-23 19:20:22.594929318 +0000 UTC"
Nov 03 18:44:20.686252 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.686247    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15072h36m1.908684987s"
Nov 03 18:44:20.740881 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:20.740855    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:44:20.746973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.746949    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:44:20.821304 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.821271    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:20.882981 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.882954    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:44:20.885394 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.885380    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:44:20.947897 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:20.947875    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:44:21.070107 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.070079    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:44:21.070319 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.070283    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:44:21.126248 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.126216    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcbaad710a39449db7d66d55191eacf.slice/crio-6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98 WatchSource:0}: Error finding container 6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98: Status 404 returned error can't find the container with id 6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98
Nov 03 18:44:21.126612 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.126593    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b200ea252cb76e83bc321c9ddace574.slice/crio-7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853 WatchSource:0}: Error finding container 7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853: Status 404 returned error can't find the container with id 7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853
Nov 03 18:44:21.132557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.132543    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:44:21.391956 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.391870    2579 apiserver.go:52] "Watching apiserver"
Nov 03 18:44:21.421203 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.421179    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 18:44:21.497609 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.497577    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:44:21.497760 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.497639    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 18:44:21.497936 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.497918    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jv87r","kube-system/global-pull-secret-syncer-94hv8","openshift-dns/node-resolver-mfgjk","openshift-image-registry/node-ca-4xkkk","openshift-multus/multus-additional-cni-plugins-q52x4","openshift-network-diagnostics/network-check-target-7dp7q","openshift-cluster-node-tuning-operator/tuned-kx25z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-4sblg","openshift-network-operator/iptables-alerter-tddmx","openshift-ovn-kubernetes/ovnkube-node-5g7pb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v","kube-system/konnectivity-agent-ddzdz","kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"]
Nov 03 18:44:21.500407 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.500387    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.501593 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.501570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.501695 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.501631    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:21.503131 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.503111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.505819 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.505800    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.507273 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.507255    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.508835 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.508818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:21.508919 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.508879    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:21.510100 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.510086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.511475 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.511460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.511565 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.511513    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:21.512854 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.512838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.514327 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.514310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.515861 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.515846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.517600 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.517583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.522199 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.522179    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 18:44:21.533521 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.533607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.533607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5rd\" (UniqueName: \"kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.533607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fbw\" (UniqueName: \"kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.533749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.533749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.533749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.533749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533738    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.533930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.533930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.533930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.533930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.533930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.534097 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.534097 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.533973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.534097 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.534097 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.534097 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534136    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6w9\" (UniqueName: \"kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.534254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hksct\" (UniqueName: \"kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534270    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534286    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52np\" (UniqueName: \"kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534596    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534686    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvtp\" (UniqueName: \"kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.534827 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534841    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.534994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535043    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535170 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmxk\" (UniqueName: \"kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdwrd\" (UniqueName: \"kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535338    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535426    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.535514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535759    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535822    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.535930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.536282 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.536282 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.535980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sn22\" (UniqueName: \"kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.536282 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.536011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.536282 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.536040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.554109 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.554087    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 18:44:21.560451 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.560431    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\""
Nov 03 18:44:21.560645 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.560609    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"5b200ea252cb76e83bc321c9ddace574","Type":"ContainerStarted","Data":"7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853"}
Nov 03 18:44:21.560920 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.560907    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\""
Nov 03 18:44:21.560968 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.560919    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\""
Nov 03 18:44:21.561752 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.561734    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerStarted","Data":"6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98"}
Nov 03 18:44:21.622092 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622064    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 18:44:21.622255 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622237    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 18:44:21.622387 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622369    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\""
Nov 03 18:44:21.622483 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622468    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\""
Nov 03 18:44:21.622528 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622483    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\""
Nov 03 18:44:21.622528 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622513    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\""
Nov 03 18:44:21.622607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622516    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\""
Nov 03 18:44:21.622607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622517    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 18:44:21.622607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.622593    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\""
Nov 03 18:44:21.636424 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.636478 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.636478 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.636478 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636478 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmxk\" (UniqueName: \"kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636559    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636579    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636586    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.636650 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636611    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwrd\" (UniqueName: \"kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636845    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636911    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.636940 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636940    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636978    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636992    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.636942    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637148    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637216    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637233    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.637321 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637296    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637306    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637318    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637400    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637491    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637503    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637535    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637576    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637593    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sn22\" (UniqueName: \"kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.638069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5rd\" (UniqueName: \"kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637755    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fbw\" (UniqueName: \"kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.637830    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637837    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.637933    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.638000    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:22.137985897 +0000 UTC m=+3.209372163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638086    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.638557 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638131    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638157    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638175    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638213    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638260    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638289    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6w9\" (UniqueName: \"kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638322    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638347    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.638374    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:21.639268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.638415    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:22.138401136 +0000 UTC m=+3.209787412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksct\" (UniqueName: \"kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638482    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638638    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52np\" (UniqueName: \"kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638645    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638669    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638711    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640143 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638781    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638805    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638860    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638916    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638949    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.638976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639003    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639028    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvtp\" (UniqueName: \"kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.640980 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639164    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639180    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639194    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639311    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639466    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639480    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.639594    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.640882    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.640899    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.641558 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.641044    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.642076 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.641411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.651425 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.651405    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 18:44:21.651507 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.651457    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7gt2n"
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654195    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654323    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654331    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654325    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654339    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654333    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654362    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654368    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654394    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654420    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654462 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654428    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654514    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654548    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654559    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654569    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654702    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654744    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654761    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654771    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654764    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654810    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 18:44:21.654917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.654848    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 18:44:21.657251 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.657232    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.657377 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.657355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.658963 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.658947    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.659085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.659174 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659155    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.659235 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.659377 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659360    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.659444 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659429    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.659725 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.659705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.671220 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.671203    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 18:44:21.678877 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.678863    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.698219 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.697957    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:21.698219 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.697980    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:21.698219 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.698011    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:21.698219 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:21.698109    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:22.198091368 +0000 UTC m=+3.269477634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:21.700318 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.700288    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwrd\" (UniqueName: \"kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.700418 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.700361    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sn22\" (UniqueName: \"kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.700538 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.700521    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52np\" (UniqueName: \"kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.700600 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.700547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5rd\" (UniqueName: \"kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.700800 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.700769    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmxk\" (UniqueName: \"kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.701244 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.701229    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksct\" (UniqueName: \"kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.701294 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.701249    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvtp\" (UniqueName: \"kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.701838 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.701817    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6w9\" (UniqueName: \"kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.702103 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.702087    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fbw\" (UniqueName: \"kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:21.812755 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.812726    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jv87r"
Nov 03 18:44:21.817933 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.817908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:44:21.821559 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.821533    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77675da6_c438_43cd_aa5a_64bc029c93c3.slice/crio-8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565 WatchSource:0}: Error finding container 8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565: Status 404 returned error can't find the container with id 8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565
Nov 03 18:44:21.824769 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.824750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:44:21.825868 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.825849    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3d8351_d66e_4646_93e1_d6d5245c5f65.slice/crio-2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a WatchSource:0}: Error finding container 2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a: Status 404 returned error can't find the container with id 2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a
Nov 03 18:44:21.832471 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.832453    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f511080_dd62_473c_b8d6_14ff2c0aa23d.slice/crio-07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9 WatchSource:0}: Error finding container 07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9: Status 404 returned error can't find the container with id 07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9
Nov 03 18:44:21.836487 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.836464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:44:21.837541 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.837529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:44:21.841876 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.841854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:44:21.846429 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.846409    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92594b38_de85_4648_ba98_f27a2da216b9.slice/crio-f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c WatchSource:0}: Error finding container f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c: Status 404 returned error can't find the container with id f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c
Nov 03 18:44:21.847036 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.846979    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94db9f0f_6fb5_4aea_bbf7_0c3afa685fb0.slice/crio-fd6528316b61f41a8c7193a303bb60e39d502ef27a44dd7675eb342b2bf0d169 WatchSource:0}: Error finding container fd6528316b61f41a8c7193a303bb60e39d502ef27a44dd7675eb342b2bf0d169: Status 404 returned error can't find the container with id fd6528316b61f41a8c7193a303bb60e39d502ef27a44dd7675eb342b2bf0d169
Nov 03 18:44:21.850354 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.850334    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513d0420_ea97_401f_ad3a_9ee14c44a819.slice/crio-221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09 WatchSource:0}: Error finding container 221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09: Status 404 returned error can't find the container with id 221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09
Nov 03 18:44:21.859874 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.859855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:21.862080 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.862057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:21.863502 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:21.863475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:44:21.867742 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.867720    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9aec1c_d239_4cd6_b8f6_c6000e4cb6b0.slice/crio-1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527 WatchSource:0}: Error finding container 1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527: Status 404 returned error can't find the container with id 1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527
Nov 03 18:44:21.870908 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.870889    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89884da1_8f1b_4d40_a0ca_9d168399a458.slice/crio-b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179 WatchSource:0}: Error finding container b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179: Status 404 returned error can't find the container with id b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179
Nov 03 18:44:21.873582 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:21.873565    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8326c46d_81bd_4d7f_801b_8a6aa2c9e724.slice/crio-10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616 WatchSource:0}: Error finding container 10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616: Status 404 returned error can't find the container with id 10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616
Nov 03 18:44:22.143657 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.143619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:22.143860 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.143675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:22.143860 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.143848    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:22.143979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.143893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:23.143880778 +0000 UTC m=+4.215267054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:22.143979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.143935    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:22.143979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.143958    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:23.143949735 +0000 UTC m=+4.215336010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:22.244920 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.244887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:22.245113 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.245084    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:22.245206 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.245117    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:22.245206 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.245130    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:22.245206 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:22.245185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:23.245171538 +0000 UTC m=+4.316557814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:22.565423 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.565390    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tddmx" event={"ID":"513d0420-ea97-401f-ad3a-9ee14c44a819","Type":"ContainerStarted","Data":"221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09"}
Nov 03 18:44:22.566909 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.566880    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kx25z" event={"ID":"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0","Type":"ContainerStarted","Data":"fd6528316b61f41a8c7193a303bb60e39d502ef27a44dd7675eb342b2bf0d169"}
Nov 03 18:44:22.568497 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.568466    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xkkk" event={"ID":"2f511080-dd62-473c-b8d6-14ff2c0aa23d","Type":"ContainerStarted","Data":"07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9"}
Nov 03 18:44:22.569763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.569742    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mfgjk" event={"ID":"ce3d8351-d66e-4646-93e1-d6d5245c5f65","Type":"ContainerStarted","Data":"2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a"}
Nov 03 18:44:22.571302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.571275    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jv87r" event={"ID":"77675da6-c438-43cd-aa5a-64bc029c93c3","Type":"ContainerStarted","Data":"8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565"}
Nov 03 18:44:22.572614 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.572585    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v" event={"ID":"8326c46d-81bd-4d7f-801b-8a6aa2c9e724","Type":"ContainerStarted","Data":"10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616"}
Nov 03 18:44:22.573524 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.573506    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddzdz" event={"ID":"89884da1-8f1b-4d40-a0ca-9d168399a458","Type":"ContainerStarted","Data":"b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179"}
Nov 03 18:44:22.574588 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.574570    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527"}
Nov 03 18:44:22.575652 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.575634    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerStarted","Data":"f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c"}
Nov 03 18:44:22.655680 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.655614    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:21 +0000 UTC" deadline="2028-06-21 08:41:21.333426526 +0000 UTC"
Nov 03 18:44:22.655680 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:22.655675    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14293h56m58.677756365s"
Nov 03 18:44:23.151370 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.151182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:23.151370 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.151234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:23.151640 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.151393    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:23.151640 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.151463    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:25.15144399 +0000 UTC m=+6.222830267 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:23.151998 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.151893    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:23.151998 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.151973    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:25.151956153 +0000 UTC m=+6.223342431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:23.252733 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.252333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:23.252733 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.252595    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:23.252733 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.252616    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:23.252733 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.252630    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:23.252733 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.252698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:25.252679243 +0000 UTC m=+6.324065509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:23.556885 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.556854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:23.557052 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.557025    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:23.557129 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.557114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:23.557207 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.557189    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:23.557281 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.557270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:23.558031 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:23.557449    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:23.656286 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.656204    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:21 +0000 UTC" deadline="2028-06-13 05:24:43.123391407 +0000 UTC"
Nov 03 18:44:23.656286 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:23.656272    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14098h40m19.467123049s"
Nov 03 18:44:24.593606 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:24.593262    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"5b200ea252cb76e83bc321c9ddace574","Type":"ContainerStarted","Data":"872561eceeb967a7e743daff53fb45849d81d85313113e625deeaee607764c16"}
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.176803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.176862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.176922    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.176996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.176975567 +0000 UTC m=+10.248361837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.177000    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:25.177070 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.177051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.177036497 +0000 UTC m=+10.248422782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:25.278031 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.277985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:25.278226 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.278209    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:25.278321 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.278232    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:25.278321 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.278244    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:25.278321 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.278300    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:29.278281213 +0000 UTC m=+10.349667494 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:25.555128 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.554913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:25.555128 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.555049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:25.555128 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.555049    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:25.555405 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.555181    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:25.555405 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.555212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:25.555405 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:25.555273    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:25.601023 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.600957    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="dbcc31f7222a228bdf84b9894b4004df82ab08602c4a4d1bfe4dba7e7ffd7582" exitCode=0
Nov 03 18:44:25.601663 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.601639    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"dbcc31f7222a228bdf84b9894b4004df82ab08602c4a4d1bfe4dba7e7ffd7582"}
Nov 03 18:44:25.605069 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.605045    2579 generic.go:356] "Generic (PLEG): container finished" podID="8bcbaad710a39449db7d66d55191eacf" containerID="226db2a07d8acd618382bf4337dc41ed7284f5535114067958a58fe85f483606" exitCode=0
Nov 03 18:44:25.605896 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.605876    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerDied","Data":"226db2a07d8acd618382bf4337dc41ed7284f5535114067958a58fe85f483606"}
Nov 03 18:44:25.733845 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:25.733672    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal" podStartSLOduration=5.733652477 podStartE2EDuration="5.733652477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:24.710917151 +0000 UTC m=+5.782303437" watchObservedRunningTime="2026-11-03 18:44:25.733652477 +0000 UTC m=+6.805038774"
Nov 03 18:44:26.612196 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:26.612158    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerStarted","Data":"a2357aa3362e69d15abc9d68fbaef8e155d844da2573f2a4755a43424eb63770"}
Nov 03 18:44:26.620390 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:26.619997    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ddzdz" event={"ID":"89884da1-8f1b-4d40-a0ca-9d168399a458","Type":"ContainerStarted","Data":"34910774398c18769a479b19badd68846a6e49a66be7904bf34be9deff2937a7"}
Nov 03 18:44:26.730262 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:26.730179    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" podStartSLOduration=6.730159291 podStartE2EDuration="6.730159291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:26.729096954 +0000 UTC m=+7.800483240" watchObservedRunningTime="2026-11-03 18:44:26.730159291 +0000 UTC m=+7.801545577"
Nov 03 18:44:27.554999 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:27.554963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:27.554999 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:27.554987    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:27.555226 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:27.555128    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:27.555226 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:27.555165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:27.555328 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:27.555238    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:27.555328 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:27.555318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:27.626773 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:27.626734    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xkkk" event={"ID":"2f511080-dd62-473c-b8d6-14ff2c0aa23d","Type":"ContainerStarted","Data":"b348dc5afafa462bae98ce4d2fbd3e8cab03b05e8fef9c3dc55e2be208e78ed7"}
Nov 03 18:44:27.743378 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:27.743322    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ddzdz" podStartSLOduration=3.392915007 podStartE2EDuration="7.743304265s" totalImagesPullingTime="4.350389258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:26.8495596 +0000 UTC m=+7.920945890" watchObservedRunningTime="2026-11-03 18:44:27.743304265 +0000 UTC m=+8.814690551"
Nov 03 18:44:28.630935 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:28.630880    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v" event={"ID":"8326c46d-81bd-4d7f-801b-8a6aa2c9e724","Type":"ContainerStarted","Data":"cf19563db6dd2e2ebf8dad9f7a61b5232f10cd2b00298fbb154cc3f1a19a1148"}
Nov 03 18:44:29.215913 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.215848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:29.215913 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.215911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:29.216189 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.216056    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:29.216189 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.216114    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:37.216095706 +0000 UTC m=+18.287481974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:29.216455 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.216435    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:29.216529 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.216479    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:37.21647014 +0000 UTC m=+18.287856403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:29.316819 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.316765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:29.316979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.316945    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:29.316979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.316965    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:29.316979 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.316978    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:29.317091 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.317040    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:37.317022961 +0000 UTC m=+18.388409228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.554547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.554676    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.555069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.555172    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:29.555234    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:29.555434 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:29.555305    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:30.358505 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.358476    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 18:44:30.487847 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.487644    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:44:30.358500122Z","UUID":"bdad66ee-8822-45bc-871b-7b4390d4142e","Handler":null,"Name":"","Endpoint":""}
Nov 03 18:44:30.490021 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.489547    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 18:44:30.490021 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.489584    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 18:44:30.638181 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.638141    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v" event={"ID":"8326c46d-81bd-4d7f-801b-8a6aa2c9e724","Type":"ContainerStarted","Data":"57ace88369d680a64fb3590b1fef0373d42ea040e8b2dcb8b03da59e9293c436"}
Nov 03 18:44:30.641080 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.640189    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jv87r" event={"ID":"77675da6-c438-43cd-aa5a-64bc029c93c3","Type":"ContainerStarted","Data":"daad8162ff2e3e8cd73020f3eb4be73417c566932f0972cd9693162cd5e65a7c"}
Nov 03 18:44:30.687404 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.687157    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:30.688200 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.688117    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:30.762648 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.762592    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jv87r" podStartSLOduration=2.637778697 podStartE2EDuration="10.762576278s" totalImagesPullingTime="8.124797581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:30.762167702 +0000 UTC m=+11.833553991" watchObservedRunningTime="2026-11-03 18:44:30.762576278 +0000 UTC m=+11.833962566"
Nov 03 18:44:30.763001 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:30.762959    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xkkk" podStartSLOduration=5.618487544 podStartE2EDuration="10.762950658s" totalImagesPullingTime="5.144463114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:27.74404036 +0000 UTC m=+8.815426648" watchObservedRunningTime="2026-11-03 18:44:30.762950658 +0000 UTC m=+11.834336944"
Nov 03 18:44:31.554861 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.554644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:31.555319 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.554666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:31.555319 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.554674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:31.555319 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:31.554992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:31.555319 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:31.555098    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:31.555493 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:31.555456    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:31.645737 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.644952    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="f7a3f32de0a43a89b39a6116637bef818d83da71face623b8401c14f157fac6a" exitCode=0
Nov 03 18:44:31.645737 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.645027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"f7a3f32de0a43a89b39a6116637bef818d83da71face623b8401c14f157fac6a"}
Nov 03 18:44:31.650119 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.650091    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mfgjk" event={"ID":"ce3d8351-d66e-4646-93e1-d6d5245c5f65","Type":"ContainerStarted","Data":"92833c07c05362457ed88a1091b73f9c6dbf344b5bc052856a0c06744b9ef1e4"}
Nov 03 18:44:31.904593 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:31.904486    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mfgjk" podStartSLOduration=2.897536893 podStartE2EDuration="11.904467049s" totalImagesPullingTime="9.006930156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:31.904036384 +0000 UTC m=+12.975422671" watchObservedRunningTime="2026-11-03 18:44:31.904467049 +0000 UTC m=+12.975853335"
Nov 03 18:44:32.654752 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:32.654710    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v" event={"ID":"8326c46d-81bd-4d7f-801b-8a6aa2c9e724","Type":"ContainerStarted","Data":"a88d8ca03167577b7bfa5cd2c34c18929deba1e1fe3bf4929f73fbe070ccec76"}
Nov 03 18:44:32.656512 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:32.656485    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tddmx" event={"ID":"513d0420-ea97-401f-ad3a-9ee14c44a819","Type":"ContainerStarted","Data":"4efd8ef12330cf81cc7b56c0faef5c28a694325c2d60a19225004f2371165f28"}
Nov 03 18:44:32.658018 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:32.657993    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kx25z" event={"ID":"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0","Type":"ContainerStarted","Data":"3562eb21c471710b6278312cd1e1a23f179b15d6be728796fc02d433e41efef0"}
Nov 03 18:44:32.775011 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:32.774912    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v" podStartSLOduration=3.06251745 podStartE2EDuration="12.774893758s" totalImagesPullingTime="9.712376308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:32.774303544 +0000 UTC m=+13.845689830" watchObservedRunningTime="2026-11-03 18:44:32.774893758 +0000 UTC m=+13.846280046"
Nov 03 18:44:32.894007 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:32.893950    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kx25z" podStartSLOduration=2.231909939 podStartE2EDuration="12.893929639s" totalImagesPullingTime="10.6620197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:32.893335591 +0000 UTC m=+13.964721875" watchObservedRunningTime="2026-11-03 18:44:32.893929639 +0000 UTC m=+13.965315925"
Nov 03 18:44:33.010222 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.010169    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tddmx" podStartSLOduration=4.784961449 podStartE2EDuration="13.010154085s" totalImagesPullingTime="8.225192636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:33.009708609 +0000 UTC m=+14.081094895" watchObservedRunningTime="2026-11-03 18:44:33.010154085 +0000 UTC m=+14.081540371"
Nov 03 18:44:33.557900 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.557869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:33.558064 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.557921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:33.558064 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.558008    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:33.558064 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:33.558024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:33.558184 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:33.558122    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:33.558227 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:33.558209    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:33.662594 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.662561    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="daca2c8bc9925f1f134e3877336867c96dfdcbace156bad5b30d5f7c8c896013" exitCode=0
Nov 03 18:44:33.663212 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:33.662688    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"daca2c8bc9925f1f134e3877336867c96dfdcbace156bad5b30d5f7c8c896013"}
Nov 03 18:44:35.556749 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:35.556551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:35.557285 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:35.556551    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:35.557285 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:35.556865    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:35.557285 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:35.556619    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:35.557285 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:35.556989    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:35.557285 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:35.557070    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:35.669442 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:35.669409    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="e5e15f3ac717adf819a6204cf3c6d524e0d0d29fb2c9b2de9082f1ae06b72ed6" exitCode=0
Nov 03 18:44:35.669608 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:35.669477    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"e5e15f3ac717adf819a6204cf3c6d524e0d0d29fb2c9b2de9082f1ae06b72ed6"}
Nov 03 18:44:36.674855 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.674829    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:44:36.675283 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675249    2579 generic.go:356] "Generic (PLEG): container finished" podID="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" containerID="294ffb7ce89b4dfe72acdb7b2e5d709c62d5e0e350f6e5594db77a1380f7b859" exitCode=1
Nov 03 18:44:36.675340 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675295    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"4de5de4aae6ebad5be383e109441ad366d7ae6aa2572e27060e749d92a1ea4be"}
Nov 03 18:44:36.675403 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675386    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"24d87b646fd031fcb6edaababbbfb57a05c7e6a2d3c6fed2797cb52db525b854"}
Nov 03 18:44:36.675460 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675448    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"cb4c4155aca48cd4cb7567290e8e847d2158f77cda580bbe1c31cdc687e811a5"}
Nov 03 18:44:36.675509 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675498    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerDied","Data":"294ffb7ce89b4dfe72acdb7b2e5d709c62d5e0e350f6e5594db77a1380f7b859"}
Nov 03 18:44:36.675556 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:36.675548    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"3454a317f7ee972aa3b837c6488e390c6f2adb8df53790cb32c6d769f8442ce6"}
Nov 03 18:44:37.281917 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.281878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:37.282087 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.281922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:37.282087 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.282048    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:37.282192 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.282099    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:37.282192 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.282114    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret podName:bca87fa3-f076-4255-9cb1-6a93dac8ff63 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:53.282097882 +0000 UTC m=+34.353484164 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret") pod "global-pull-secret-syncer-94hv8" (UID: "bca87fa3-f076-4255-9cb1-6a93dac8ff63") : object "kube-system"/"original-pull-secret" not registered
Nov 03 18:44:37.282192 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.282146    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs podName:86362a0d-cbf8-4478-b304-f6de2fd0bb0f nodeName:}" failed. No retries permitted until 2026-11-03 18:44:53.282133233 +0000 UTC m=+34.353519501 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs") pod "network-metrics-daemon-4sblg" (UID: "86362a0d-cbf8-4478-b304-f6de2fd0bb0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 18:44:37.382984 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.382932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:37.383151 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.383124    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 18:44:37.383151 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.383146    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 18:44:37.383271 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.383158    2579 projected.go:196] Error preparing data for projected volume kube-api-access-jr7qb for pod openshift-network-diagnostics/network-check-target-7dp7q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:37.383271 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.383226    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb podName:777bcab9-01dd-4932-8c5d-0f623570e177 nodeName:}" failed. No retries permitted until 2026-11-03 18:44:53.383207281 +0000 UTC m=+34.454593559 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jr7qb" (UniqueName: "kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb") pod "network-check-target-7dp7q" (UID: "777bcab9-01dd-4932-8c5d-0f623570e177") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 18:44:37.466659 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.466611    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:37.554124 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.554039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:37.554265 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.554041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:37.554265 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.554187    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:37.554265 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.554042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:37.554265 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.554243    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:37.554464 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:37.554293    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:37.679950 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.679926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:44:37.680428 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.680294    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"a1b896001e9936e59fd3b9bbb975ce5fc78e3cb0e496c6c544f90f11d1e1f0ba"}
Nov 03 18:44:37.683144 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.683112    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerStarted","Data":"1d74cdf8ba23ff5e53870c6888ce96158b9b67127ce685684b362e9235cf0990"}
Nov 03 18:44:37.759613 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:37.759581    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:38.687718 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:38.687687    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="1d74cdf8ba23ff5e53870c6888ce96158b9b67127ce685684b362e9235cf0990" exitCode=0
Nov 03 18:44:38.688087 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:38.687733    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"1d74cdf8ba23ff5e53870c6888ce96158b9b67127ce685684b362e9235cf0990"}
Nov 03 18:44:39.554652 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.554617    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:39.554831 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.554746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:39.554831 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:39.554767    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:39.554831 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:39.554811    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:39.555059 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.554878    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:39.555059 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:39.554990    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:39.692972 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.692945    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:44:39.693444 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.693313    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"cdcd853def4bcc1809eccb9ffa9d6fc9d108429e4a6a08c5934ba688c44621e3"}
Nov 03 18:44:39.695994 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.695970    2579 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="e0e68128b62d6e482260a26b858e3dc846764de23e30f99d8f52301e65fda4a4" exitCode=0
Nov 03 18:44:39.696109 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:39.696011    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerDied","Data":"e0e68128b62d6e482260a26b858e3dc846764de23e30f99d8f52301e65fda4a4"}
Nov 03 18:44:40.097256 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:40.097223    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:40.348632 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:40.348547    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:40.597435 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:40.597403    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:40.701477 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:40.701404    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q52x4" event={"ID":"92594b38-de85-4648-ba98-f27a2da216b9","Type":"ContainerStarted","Data":"0489fe25aac1017274dd9e30234cd7d47cabe867e4b5e1f67999bb90ab49ebf5"}
Nov 03 18:44:40.826806 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:40.826741    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q52x4" podStartSLOduration=7.578914627 podStartE2EDuration="20.826726276s" totalImagesPullingTime="13.247811649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:40.82609525 +0000 UTC m=+21.897481535" watchObservedRunningTime="2026-11-03 18:44:40.826726276 +0000 UTC m=+21.898112561"
Nov 03 18:44:41.422939 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.422729    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:41.554499 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.554472    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:41.554814 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.554526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:41.554814 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.554533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:41.554814 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:41.554624    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:41.554814 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:41.554735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:41.555042 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:41.554831    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:41.707224 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.707144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:44:41.707686 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.707459    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"ad4611b917b97675e94edbfd6bbb5e8d8d54f3625fd2184544ad8cbea4a3bdc3"}
Nov 03 18:44:41.708028 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.708009    2579 scope.go:122] "RemoveContainer" containerID="294ffb7ce89b4dfe72acdb7b2e5d709c62d5e0e350f6e5594db77a1380f7b859"
Nov 03 18:44:41.978660 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:41.978592    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:42.229200 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.229112    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:42.713278 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.713254    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:44:42.713834 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.713619    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" event={"ID":"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0","Type":"ContainerStarted","Data":"9344110ce30b02a5b070e58b616e115a366c67de2f963c9a45ddb01258044289"}
Nov 03 18:44:42.713834 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.713756    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:44:42.714047 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.714029    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:42.714114 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.714104    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:42.728674 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.728654    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:42.728784 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.728745    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:42.998818 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:42.998756    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" podStartSLOduration=8.675340049 podStartE2EDuration="22.998741368s" totalImagesPullingTime="14.323401319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:42.998351985 +0000 UTC m=+24.069738272" watchObservedRunningTime="2026-11-03 18:44:42.998741368 +0000 UTC m=+24.070127683"
Nov 03 18:44:43.553987 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:43.553957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:43.553987 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:43.553978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:43.554174 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:43.554064    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:43.554174 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:43.554131    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:43.554234 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:43.554213    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:43.554281 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:43.554260    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:43.715570 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:43.715546    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:44:44.470670 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:44.470639    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:44.470890 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:44.470761    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:44:44.471179 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:44.471161    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:44:44.717352 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:44.717328    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:44:45.224865 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.224813    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:45.479164 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.479083    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:45.554191 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.554156    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:45.554191 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.554176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:45.554401 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.554268    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:45.554401 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:45.554275    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:45.554401 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:45.554355    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:45.554505 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:45.554443    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:45.890314 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:45.890273    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:46.139803 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.139762    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:46.598426 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.598236    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-94hv8"]
Nov 03 18:44:46.598426 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.598365    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:46.598666 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:46.598474    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:46.601203 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.601176    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7dp7q"]
Nov 03 18:44:46.601339 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.601280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:46.601404 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:46.601380    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:46.601759 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.601739    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sblg"]
Nov 03 18:44:46.601851 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:46.601840    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:46.601940 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:46.601924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:47.107362 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:47.107324    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:47.192949 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:47.192913    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:44:47.193137 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:47.193125    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:44:47.204163 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:47.204125    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" podUID="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 18:44:47.212740 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:47.212707    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb" podUID="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" containerName="ovnkube-controller" probeResult="failure" output=""
Nov 03 18:44:48.554701 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:48.554526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:48.555133 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:48.554545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:48.555133 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:48.554608    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:48.555133 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:48.554768    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:48.555133 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:48.554934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:48.555133 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:48.555020    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:49.974130 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:49.974018    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:50.243634 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.243604    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:50.489697 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.489667    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:50.554251 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.554178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:50.554251 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.554241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:50.554399 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.554266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:50.554399 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:50.554349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7dp7q" podUID="777bcab9-01dd-4932-8c5d-0f623570e177"
Nov 03 18:44:50.554399 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:50.554381    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-94hv8" podUID="bca87fa3-f076-4255-9cb1-6a93dac8ff63"
Nov 03 18:44:50.554489 ip-10-0-137-192 kubenswrapper[2579]: E1103 18:44:50.554473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sblg" podUID="86362a0d-cbf8-4478-b304-f6de2fd0bb0f"
Nov 03 18:44:50.749244 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:50.749214    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:52.020753 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.020724    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:52.191976 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.191952    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeReady"
Nov 03 18:44:52.192130 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.192067    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 18:44:52.280341 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.280243    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:52.310233 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.310206    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qvss5"]
Nov 03 18:44:52.313085 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.313070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.361232 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.361208    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l8qkk"]
Nov 03 18:44:52.365264 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.365242    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qvss5"]
Nov 03 18:44:52.365368 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.365350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.367353 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.367336    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 18:44:52.367489 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.367475    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\""
Nov 03 18:44:52.367984 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.367972    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 18:44:52.368058 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.368046    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 18:44:52.399231 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.399204    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mq4br"]
Nov 03 18:44:52.402091 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.402072    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.413033 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.413008    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8qkk"]
Nov 03 18:44:52.458273 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.458245    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mq4br"]
Nov 03 18:44:52.467576 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467559    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 18:44:52.467702 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467687    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 18:44:52.467808 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467764    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 18:44:52.467808 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467773    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 18:44:52.467808 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467773    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\""
Nov 03 18:44:52.467808 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467764    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 18:44:52.468013 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467822    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 18:44:52.468013 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.467890    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\""
Nov 03 18:44:52.488546 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.488628 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.488628 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.488628 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccslj\" (UniqueName: \"kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.488733 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.488733 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.488733 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.488833 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.488833 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.488893 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlht\" (UniqueName: \"kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.488924 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.488898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jx9k\" (UniqueName: \"kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.554051 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.553983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:52.554157 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.554052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:52.554157 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.554079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:52.589488 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlht\" (UniqueName: \"kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.589574 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx9k\" (UniqueName: \"kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.589574 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.589574 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.589574 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.589763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccslj\" (UniqueName: \"kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.589763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.589763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.589763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.589763 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.590071 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589892    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.590071 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.590071 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.590071 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.589968    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.590475 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.590454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.590580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.590562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.593372 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.593348    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.593372 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.593364    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.593607 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.593587    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.608046 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.607999    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 18:44:52.608247 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608227    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 18:44:52.608323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608253    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 18:44:52.608323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608256    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\""
Nov 03 18:44:52.608323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608267    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 18:44:52.608323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608270    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\""
Nov 03 18:44:52.608323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.608259    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\""
Nov 03 18:44:52.652090 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.652062    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx9k\" (UniqueName: \"kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.652177 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.652107    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlht\" (UniqueName: \"kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.652177 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.652151    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccslj\" (UniqueName: \"kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:52.679769 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.679746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:44:52.715771 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.715743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:52.923726 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:52.923645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:44:53.124973 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.124938    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mq4br"]
Nov 03 18:44:53.127838 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.127817    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l8qkk"]
Nov 03 18:44:53.128985 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.128962    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c700ed_1257_476f_a0e8_2a7483d06af7.slice/crio-c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964 WatchSource:0}: Error finding container c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964: Status 404 returned error can't find the container with id c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964
Nov 03 18:44:53.130876 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.130853    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20f1d5f_64dc_42af_bed3_167aab7074ee.slice/crio-3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199 WatchSource:0}: Error finding container 3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199: Status 404 returned error can't find the container with id 3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199
Nov 03 18:44:53.295999 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.295962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:53.296180 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.296153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:53.298373 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.298351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:53.298471 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.298373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:53.300814 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.300748    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qvss5"]
Nov 03 18:44:53.305022 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.305004    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a6bf43_66d1_4875_9821_58d2a9ff008d.slice/crio-ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784 WatchSource:0}: Error finding container ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784: Status 404 returned error can't find the container with id ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784
Nov 03 18:44:53.396838 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.396806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:53.399467 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.399448    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:53.467213 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.467142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:44:53.472311 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.472289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:53.476411 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.476397    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:44:53.737920 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.737838    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq4br" event={"ID":"d5c700ed-1257-476f-a0e8-2a7483d06af7","Type":"ContainerStarted","Data":"c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964"}
Nov 03 18:44:53.739401 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.739366    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8qkk" event={"ID":"e20f1d5f-64dc-42af-bed3-167aab7074ee","Type":"ContainerStarted","Data":"5a73fe48b7c1a0480d8f2b3d0a626184d653b220ee20b09d3ed5cdb1b202a05f"}
Nov 03 18:44:53.739528 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.739432    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8qkk" event={"ID":"e20f1d5f-64dc-42af-bed3-167aab7074ee","Type":"ContainerStarted","Data":"3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199"}
Nov 03 18:44:53.740531 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.740507    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qvss5" event={"ID":"08a6bf43-66d1-4875-9821-58d2a9ff008d","Type":"ContainerStarted","Data":"ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784"}
Nov 03 18:44:53.860054 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.859991    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sblg"]
Nov 03 18:44:53.863272 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.863246    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86362a0d_cbf8_4478_b304_f6de2fd0bb0f.slice/crio-ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d WatchSource:0}: Error finding container ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d: Status 404 returned error can't find the container with id ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d
Nov 03 18:44:53.865723 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.865696    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:53.910217 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.910184    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-94hv8"]
Nov 03 18:44:53.913687 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:53.913664    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7dp7q"]
Nov 03 18:44:53.913924 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.913894    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca87fa3_f076_4255_9cb1_6a93dac8ff63.slice/crio-5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4 WatchSource:0}: Error finding container 5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4: Status 404 returned error can't find the container with id 5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4
Nov 03 18:44:53.916935 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:53.916870    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777bcab9_01dd_4932_8c5d_0f623570e177.slice/crio-67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9 WatchSource:0}: Error finding container 67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9: Status 404 returned error can't find the container with id 67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9
Nov 03 18:44:54.112094 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:54.112059    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:54.744367 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:54.744330    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-94hv8" event={"ID":"bca87fa3-f076-4255-9cb1-6a93dac8ff63","Type":"ContainerStarted","Data":"5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4"}
Nov 03 18:44:54.745930 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:54.745901    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sblg" event={"ID":"86362a0d-cbf8-4478-b304-f6de2fd0bb0f","Type":"ContainerStarted","Data":"ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d"}
Nov 03 18:44:54.747116 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:54.747095    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7dp7q" event={"ID":"777bcab9-01dd-4932-8c5d-0f623570e177","Type":"ContainerStarted","Data":"67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9"}
Nov 03 18:44:55.758120 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:55.758081    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8qkk" event={"ID":"e20f1d5f-64dc-42af-bed3-167aab7074ee","Type":"ContainerStarted","Data":"3c85261e346fbcc815a9bba85af414bd4004b4cd1f555e04bddce7a84ba5ca13"}
Nov 03 18:44:56.137333 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:56.137236    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:56.383110 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:56.383032    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:57.338286 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.338256    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:57.592646 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.592580    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:57.768164 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.768086    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sblg" event={"ID":"86362a0d-cbf8-4478-b304-f6de2fd0bb0f","Type":"ContainerStarted","Data":"5271163e4a384bd14429e71d239136e02e3c463e460d95c97e1bacc97f801040"}
Nov 03 18:44:57.768164 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.768151    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sblg" event={"ID":"86362a0d-cbf8-4478-b304-f6de2fd0bb0f","Type":"ContainerStarted","Data":"08a72c56d81530a7175bc8f5520c843754d4965d446631f2a22bd787638ef633"}
Nov 03 18:44:57.772336 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.772290    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq4br" event={"ID":"d5c700ed-1257-476f-a0e8-2a7483d06af7","Type":"ContainerStarted","Data":"edc1a7538a8f91dddf6465673fae227849a81ed532a8dc08244b5fcb9fa294fc"}
Nov 03 18:44:57.772446 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.772342    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mq4br" event={"ID":"d5c700ed-1257-476f-a0e8-2a7483d06af7","Type":"ContainerStarted","Data":"b70516e2d0bf02d03026c688667ded5c18922373b05c8bd72b3b3a51c2d51536"}
Nov 03 18:44:57.772603 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.772577    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mq4br"
Nov 03 18:44:57.877092 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.877031    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:57.887483 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:57.887425    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4sblg" podStartSLOduration=35.11127705 podStartE2EDuration="37.887406232s" totalImagesPullingTime="2.776129182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:57.886934772 +0000 UTC m=+38.958321057" watchObservedRunningTime="2026-11-03 18:44:57.887406232 +0000 UTC m=+38.958792518"
Nov 03 18:44:58.006610 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.006544    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-mq4br" podStartSLOduration=1.883441513 podStartE2EDuration="6.006526163s" totalImagesPullingTime="4.12308465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:58.00577704 +0000 UTC m=+39.077163326" watchObservedRunningTime="2026-11-03 18:44:58.006526163 +0000 UTC m=+39.077912449"
Nov 03 18:44:58.153323 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.153232    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:58.367754 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.367729    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"]
Nov 03 18:44:58.390696 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.390675    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.404828 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.404767    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:58.430445 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.430278    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"]
Nov 03 18:44:58.445724 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.445704    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 18:44:58.458454 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.458420    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 18:44:58.480725 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.479868    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qnvrx\""
Nov 03 18:44:58.480725 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.480093    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 18:44:58.496888 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.496824    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 18:44:58.496888 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.496881    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 18:44:58.497969 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.497945    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gd5jv"]
Nov 03 18:44:58.513383 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.513352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.531770 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.531742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.531936 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.531827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.531936 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.531861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.531936 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.531887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pnz\" (UniqueName: \"kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.567759 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.567712    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 18:44:58.567920 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.567826    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 18:44:58.567920 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.567896    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltsf8\""
Nov 03 18:44:58.633112 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6tv7\" (UniqueName: \"kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633302 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.633584 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.633451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pnz\" (UniqueName: \"kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.634780 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.634750    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.637022 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.636972    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.637119 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.637038    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.639580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.639557    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 18:44:58.652180 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.652152    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:58.695717 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.695642    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pnz\" (UniqueName: \"kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.734514 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tv7\" (UniqueName: \"kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734623    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.734676 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735009 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735009 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734735    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735009 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.734891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735111 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.735082    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735343 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.735323    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.735400 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.735377    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.736906 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.736886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.737009 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.736994    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.768075 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.768050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:44:58.778527 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.778497    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qvss5" event={"ID":"08a6bf43-66d1-4875-9821-58d2a9ff008d","Type":"ContainerStarted","Data":"2cea334caa25ebdcbb651f39c767a7257665b8a72e4ec59036e9519985098ee7"}
Nov 03 18:44:58.780761 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.780730    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l8qkk" event={"ID":"e20f1d5f-64dc-42af-bed3-167aab7074ee","Type":"ContainerStarted","Data":"41df8995e043994ff4af9a2d91a7e65469db8de0e85535f6badf41ac2afc3551"}
Nov 03 18:44:58.796586 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.796567    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tv7\" (UniqueName: \"kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.869800 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.869761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:44:58.896757 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.896703    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qvss5" podStartSLOduration=1.853579237 podStartE2EDuration="6.896682741s" totalImagesPullingTime="5.043103504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:58.896053658 +0000 UTC m=+39.967439944" watchObservedRunningTime="2026-11-03 18:44:58.896682741 +0000 UTC m=+39.968069030"
Nov 03 18:44:58.900804 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:58.900763    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:59.016180 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.016128    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l8qkk" podStartSLOduration=1.963934657 podStartE2EDuration="7.016113151s" totalImagesPullingTime="5.052178494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:44:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:59.015840537 +0000 UTC m=+40.087226819" watchObservedRunningTime="2026-11-03 18:44:59.016113151 +0000 UTC m=+40.087499435"
Nov 03 18:44:59.086354 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:59.086325    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda592af2a_0e6b_4455_b71f_bf6c8fe5a14c.slice/crio-dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73 WatchSource:0}: Error finding container dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73: Status 404 returned error can't find the container with id dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73
Nov 03 18:44:59.150189 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.150146    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:59.210896 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.210861    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"]
Nov 03 18:44:59.215060 ip-10-0-137-192 kubenswrapper[2579]: W1103 18:44:59.215035    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce552fcf_36e3_435c_887d_6ceff825d956.slice/crio-d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20 WatchSource:0}: Error finding container d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20: Status 404 returned error can't find the container with id d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20
Nov 03 18:44:59.400286 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.400259    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:44:59.785477 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.785435    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd5jv" event={"ID":"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c","Type":"ContainerStarted","Data":"dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73"}
Nov 03 18:44:59.787421 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.787389    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5" event={"ID":"ce552fcf-36e3-435c-887d-6ceff825d956","Type":"ContainerStarted","Data":"f1b040d6da33fedd66e99ad7cf00e18997b15caeb106ac4b3bd3cdee513c8369"}
Nov 03 18:44:59.787540 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.787448    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5" event={"ID":"ce552fcf-36e3-435c-887d-6ceff825d956","Type":"ContainerStarted","Data":"5ee0408ef2ca7b3f4fb89ef6adfecf9c91f6f3897064098a819ebc90d35dd924"}
Nov 03 18:44:59.787540 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.787474    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5" event={"ID":"ce552fcf-36e3-435c-887d-6ceff825d956","Type":"ContainerStarted","Data":"d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20"}
Nov 03 18:44:59.789064 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.789036    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7dp7q" event={"ID":"777bcab9-01dd-4932-8c5d-0f623570e177","Type":"ContainerStarted","Data":"ea6e8077b5dc718f5269c85cf037265967929874366917529cd9abf047d99f2b"}
Nov 03 18:44:59.789657 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.789636    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:44:59.907343 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.907281    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7dp7q" podStartSLOduration=34.734389285 podStartE2EDuration="39.907261696s" totalImagesPullingTime="5.172872411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:44:59.906695117 +0000 UTC m=+40.978081402" watchObservedRunningTime="2026-11-03 18:44:59.907261696 +0000 UTC m=+40.978647988"
Nov 03 18:44:59.942527 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:44:59.942492    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:00.187354 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.187264    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:00.828654 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.828615    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 18:45:00.847065 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.847036    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.884896 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.884862    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 18:45:00.887772 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.887742    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:00.900692 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.900665    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 18:45:00.900825 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.900722    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 18:45:00.900825 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.900808    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 18:45:00.934165 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934136    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 18:45:00.934165 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934141    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 18:45:00.934294 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934196    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 18:45:00.934294 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934217    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ltsw\""
Nov 03 18:45:00.934379 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934368    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 18:45:00.934513 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.934497    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 18:45:00.958254 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958231    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958341 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958341 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958284    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958401 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958444 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958544 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958598 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958642 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958686 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958730 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958837 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958837 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958760    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:00.958837 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:00.958781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.037165 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.037137    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 18:45:01.046218 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.046192    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74994bbc58-d8mv2"]
Nov 03 18:45:01.055774 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.055756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.059232 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059232 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059232 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059232 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059392 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059392 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059392 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059392 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059392 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059531 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059531 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059531 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.059531 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.060388 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.059897    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.060388 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.060328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.061163 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.061137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.062014 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.061967    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.062425 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.062398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.062908 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.062885    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.063568 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.063542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.064075 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.064053    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.064539 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.064515    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.065152 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.065132    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.065275 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.065258    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.065550 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.065526    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.106649 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.106623    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74994bbc58-d8mv2"]
Nov 03 18:45:01.109551 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109505    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 18:45:01.109692 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109612    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 18:45:01.109692 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109634    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 18:45:01.109692 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109643    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 18:45:01.109692 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109669    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 18:45:01.109866 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109722    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jlxrp\""
Nov 03 18:45:01.109866 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.109744    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-60agudo7g89ai\""
Nov 03 18:45:01.121461 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.121440    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.131136 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.131101    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:01.160562 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n94fx\" (UniqueName: \"kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.160693 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.160765 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.160765 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.160912 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.160912 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.160853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.161150 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.161133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.161220 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.161168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.161269 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.161248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262432 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94fx\" (UniqueName: \"kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262580 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262760 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262760 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262760 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.262760 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.262723    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.263268 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.263207    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.263362 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.263294    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265190 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265161    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265281 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265401 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265376    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265571 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265549    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265616 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265580    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.265616 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.265558    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.275381 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.275363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:01.324948 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.324901    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94fx\" (UniqueName: \"kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.504508 ip-10-0-137-192 kubenswrapper[2579]: I1103 18:45:01.504271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:01.659271 ip-10-0-137-192 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 18:45:01.675173 ip-10-0-137-192 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 18:45:01.675340 ip-10-0-137-192 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 18:45:01.675386 ip-10-0-137-192 systemd[1]: kubelet.service: Consumed 2.870s CPU time.
Nov 03 18:45:01.691424 ip-10-0-137-192 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 18:45:01.788667 ip-10-0-137-192 kubenswrapper[4944]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:45:01.788667 ip-10-0-137-192 kubenswrapper[4944]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 18:45:01.788667 ip-10-0-137-192 kubenswrapper[4944]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:45:01.788667 ip-10-0-137-192 kubenswrapper[4944]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 18:45:01.790867 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790766    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.790867 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790866    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790870    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790875    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790880    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790885    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790889    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790892    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790894    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790897    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790900    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790902    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790906    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790909    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790912    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790923    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790927    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790932    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790935    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790938    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790941    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.790945 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790943    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790946    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790949    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790954    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790959    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790963    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790967    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790970    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790972    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790975    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790978    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790980    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790983    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790985    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790988    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790991    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790993    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790996    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.790999    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791001    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.791431 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791004    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791006    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791009    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791011    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791014    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791017    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791020    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791023    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791026    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791031    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791039    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791044    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791047    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791050    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791053    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791056    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791059    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791061    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791064    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.791916 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791067    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791069    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791072    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791074    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791077    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791080    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791082    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791085    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791088    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791090    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791093    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791096    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791099    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791101    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791104    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791131    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791138    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791143    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791149    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791152    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.792418 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791155    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791159    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791162    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791164    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791167    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791169    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791172    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791175    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791178    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791180    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791183    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791185    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791188    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791191    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791193    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791196    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791198    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791201    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791204    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.792908 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791206    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791209    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791211    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791214    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791217    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791219    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.791223    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792064    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792073    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792077    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792080    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792083    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792086    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792089    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792092    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792095    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792098    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792100    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792103    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.793630 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792106    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792108    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792113    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792117    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792121    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792124    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792126    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792129    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792132    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792135    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792137    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792140    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792143    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792146    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792148    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792151    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792154    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792157    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792160    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.794132 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792163    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792165    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792168    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792171    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792174    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792177    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792179    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792182    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792185    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792187    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792190    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792193    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792196    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792198    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792201    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792203    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792206    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792209    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792211    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792214    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.794585 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792216    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792219    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792222    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792225    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792229    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792232    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792236    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792238    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792241    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792244    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792246    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792249    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792254    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792256    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792259    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792262    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792265    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792267    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792270    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.795076 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792273    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792275    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792278    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792280    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792284    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792286    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792289    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792291    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792294    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792297    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792299    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792302    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792304    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792307    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792310    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792313    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792316    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792319    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792321    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.795522 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792324    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792326    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792329    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792332    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792335    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792338    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792341    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792343    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792346    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792349    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792351    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792356    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792358    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792361    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792364    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792366    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792369    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792894    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792901    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792904    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.795974 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792907    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792910    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792912    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792915    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792918    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792921    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792924    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792927    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792930    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792932    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792935    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792938    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792941    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792943    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792946    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792949    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792952    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792954    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792957    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792960    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.796449 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792962    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792965    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792967    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792972    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792975    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792978    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792980    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792983    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792985    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792988    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792991    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792996    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.792999    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793002    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793005    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793008    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793010    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793013    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793016    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.797071 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793019    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793022    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793025    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793027    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793030    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793033    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793035    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793038    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793041    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793043    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793046    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793049    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793051    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793054    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793057    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793059    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793064    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793067    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793069    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793072    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.797886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793075    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793078    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793080    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793083    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793086    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793089    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793093    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793095    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793098    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793101    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793104    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793107    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793109    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793112    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793114    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793117    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793120    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793122    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793125    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.798721 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793127    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793130    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793132    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793135    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793138    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793140    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793143    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793146    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793151    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793156    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793159    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793162    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793164    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793167    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793170    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793173    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793176    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793179    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793182    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.799565 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793185    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793188    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793191    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793193    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793196    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.793198    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793305    4944 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793316    4944 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793324    4944 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793329    4944 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793333    4944 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793337    4944 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793341    4944 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793346    4944 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793349    4944 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793352    4944 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793356    4944 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793360    4944 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793363    4944 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793366    4944 flags.go:64] FLAG: --cgroup-root=""
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793369    4944 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793372    4944 flags.go:64] FLAG: --client-ca-file=""
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793377    4944 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 18:45:01.800445 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793380    4944 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793386    4944 flags.go:64] FLAG: --cluster-domain=""
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793388    4944 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793392    4944 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793396    4944 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793399    4944 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793404    4944 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793407    4944 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793410    4944 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793414    4944 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793418    4944 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793421    4944 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793424    4944 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793427    4944 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793430    4944 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793435    4944 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793438    4944 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793442    4944 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793445    4944 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793448    4944 flags.go:64] FLAG: --enable-server="true"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793451    4944 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793457    4944 flags.go:64] FLAG: --event-burst="100"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793460    4944 flags.go:64] FLAG: --event-qps="50"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793463    4944 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 18:45:01.801443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793466    4944 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793470    4944 flags.go:64] FLAG: --eviction-hard=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793474    4944 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793477    4944 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793480    4944 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793484    4944 flags.go:64] FLAG: --eviction-soft=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793487    4944 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793491    4944 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793494    4944 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793497    4944 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793502    4944 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793505    4944 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793508    4944 flags.go:64] FLAG: --feature-gates=""
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793513    4944 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793516    4944 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793520    4944 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793523    4944 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793526    4944 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793530    4944 flags.go:64] FLAG: --help="false"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793533    4944 flags.go:64] FLAG: --hostname-override="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793537    4944 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793540    4944 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793543    4944 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 18:45:01.802351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793546    4944 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793550    4944 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793553    4944 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793556    4944 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793559    4944 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793562    4944 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793565    4944 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793568    4944 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793571    4944 flags.go:64] FLAG: --kube-reserved=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793574    4944 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793577    4944 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793580    4944 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793584    4944 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793587    4944 flags.go:64] FLAG: --lock-file=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793590    4944 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793593    4944 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793598    4944 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793604    4944 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793607    4944 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793612    4944 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793615    4944 flags.go:64] FLAG: --logging-format="text"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793618    4944 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793622    4944 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793625    4944 flags.go:64] FLAG: --manifest-url=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793627    4944 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 18:45:01.803106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793632    4944 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793636    4944 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793640    4944 flags.go:64] FLAG: --max-pods="110"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793644    4944 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793647    4944 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793650    4944 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793653    4944 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793656    4944 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793660    4944 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793663    4944 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793670    4944 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793673    4944 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793676    4944 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793679    4944 flags.go:64] FLAG: --pod-cidr=""
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793682    4944 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793685    4944 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793688    4944 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793692    4944 flags.go:64] FLAG: --port="10250"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793695    4944 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793698    4944 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0e84d7b1a247a8c13"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793701    4944 flags.go:64] FLAG: --qos-reserved=""
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793705    4944 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793708    4944 flags.go:64] FLAG: --register-node="true"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793712    4944 flags.go:64] FLAG: --register-with-taints=""
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793716    4944 flags.go:64] FLAG: --registry-burst="10"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793719    4944 flags.go:64] FLAG: --registry-qps="5"
Nov 03 18:45:01.803700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793723    4944 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793726    4944 flags.go:64] FLAG: --reserved-memory=""
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793730    4944 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793733    4944 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793736    4944 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793739    4944 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793742    4944 flags.go:64] FLAG: --runonce="false"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793745    4944 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793748    4944 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793751    4944 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793755    4944 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793758    4944 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793761    4944 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793765    4944 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793768    4944 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793771    4944 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793774    4944 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793777    4944 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793781    4944 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793784    4944 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793803    4944 flags.go:64] FLAG: --system-cgroups=""
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793807    4944 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793812    4944 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793815    4944 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793818    4944 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 18:45:01.804399 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793823    4944 flags.go:64] FLAG: --tls-min-version=""
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793826    4944 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793829    4944 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793832    4944 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793837    4944 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793840    4944 flags.go:64] FLAG: --v="2"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793844    4944 flags.go:64] FLAG: --version="false"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793850    4944 flags.go:64] FLAG: --vmodule=""
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793854    4944 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.793858    4944 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794000    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794005    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794008    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794011    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794013    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794016    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794019    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794022    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794025    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794028    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794031    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794033    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.805014 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794036    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794039    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794041    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794044    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794047    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794050    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794053    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794055    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794058    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794061    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794063    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794066    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794069    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794071    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794075    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794078    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794081    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794086    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794089    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794091    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.805678 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794094    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794097    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794099    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794102    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794105    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794107    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794110    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794113    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794115    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794121    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794125    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794128    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794131    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794134    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794137    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794140    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794143    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794146    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794149    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.806451 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794152    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794154    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794157    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794159    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794162    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794164    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794167    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794171    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794174    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794176    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794181    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794183    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794186    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794189    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794191    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794194    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794197    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794200    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794202    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794205    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.807098 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794208    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794211    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794213    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794217    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794220    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794222    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794225    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794228    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794230    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794233    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794235    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794238    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794241    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794244    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794246    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794249    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794251    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794254    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794256    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.807764 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794261    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794263    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794266    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794270    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794274    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794278    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794281    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794284    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794286    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794289    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794292    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794295    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794298    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794300    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794303    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794306    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.808318 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.794333    4944 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794368    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794372    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794375    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794378    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794381    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794384    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794386    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794389    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794392    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794395    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794397    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794401    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794405    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794408    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794411    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794414    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794419    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794422    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.808748 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794425    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794428    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794430    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794433    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794436    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794438    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794441    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794444    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794446    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794449    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794452    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794454    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794457    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794460    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794462    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794466    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794468    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794471    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794474    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794476    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.809341 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794479    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794481    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794485    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794487    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794490    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794492    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794495    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794498    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794501    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794503    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794507    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794510    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794515    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794517    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794520    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794523    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794525    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794528    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794531    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794533    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.809915 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794536    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794538    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794541    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794544    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794547    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794549    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794554    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794557    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794561    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794564    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794567    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794569    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794572    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794575    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794578    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794584    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794587    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794590    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794593    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794595    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.810381 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794598    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794601    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794605    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794608    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794611    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794614    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794617    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794620    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794622    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794625    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794628    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794630    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794633    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794635    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794638    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794641    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794645    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794647    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794650    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.810958 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794653    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794655    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794659    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794661    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794664    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794667    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794669    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794672    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.794674    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.799230    4944 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.799252    4944 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.799272    4944 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.799280    4944 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799365    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799374    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799379    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799384    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799388    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799393    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799398    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.811394 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799403    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799410    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799415    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799421    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799426    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799430    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799434    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799439    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799444    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799449    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799453    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799457    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799462    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799468    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799473    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799477    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799481    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799485    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799490    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.811886 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799495    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799500    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799504    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799509    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799513    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799518    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799522    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799526    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799531    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799535    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799539    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799544    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799548    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799552    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799556    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799561    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799565    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799569    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799573    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.812332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799577    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799581    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799585    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799590    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799594    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799599    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799604    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799609    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799614    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799619    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799624    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799628    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799632    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799636    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799641    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799646    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799651    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799656    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799661    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799665    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.812810 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799669    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799674    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799681    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799687    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799693    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799698    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799702    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799707    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799712    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799717    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799722    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799726    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799731    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799735    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799740    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799744    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799749    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799753    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799758    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.813290 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799762    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799767    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799772    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799777    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799781    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799801    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799806    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799810    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799814    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799819    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799823    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799827    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799832    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799836    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799841    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799846    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799851    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799855    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799860    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799864    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.813861 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799869    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.799873    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.799916    4944 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800154    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800162    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800168    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800174    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800178    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800183    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800187    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800192    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800196    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800200    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800204    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800209    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800214    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.814354 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800218    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800223    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800227    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800231    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800236    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800240    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800244    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800248    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800253    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800257    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800261    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800265    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800269    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800274    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800278    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800283    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800287    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800291    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800296    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800300    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.814781 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800304    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800309    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800313    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800318    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800323    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800329    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800335    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800339    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800343    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800348    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800352    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800356    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800361    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800365    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800370    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800375    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800380    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800385    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800389    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.815332 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800393    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800398    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800402    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800406    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800414    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800418    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800423    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800427    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800432    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800436    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800440    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800445    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800449    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800454    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800458    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800462    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800466    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800470    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800475    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.815782 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800479    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800484    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800488    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800493    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800497    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800501    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800506    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800510    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800514    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800519    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800524    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800528    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800532    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800539    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800544    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800549    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800554    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800560    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800565    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800569    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.816251 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800573    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800577    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800582    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800588    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800594    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800599    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800604    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800609    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800613    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800617    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800621    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800626    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800637    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800642    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800646    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.800690    4944 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 18:45:01.816728 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800741    4944 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800747    4944 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800751    4944 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800756    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800761    4944 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800765    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800770    4944 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800774    4944 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800779    4944 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800783    4944 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800802    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800806    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800811    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800817    4944 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800821    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800825    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800829    4944 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800834    4944 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800838    4944 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800842    4944 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 18:45:01.817167 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800847    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800851    4944 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800855    4944 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800860    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800864    4944 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800868    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800872    4944 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800876    4944 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800880    4944 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800884    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800889    4944 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800893    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800897    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800901    4944 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800905    4944 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800910    4944 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800914    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800919    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800923    4944 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 18:45:01.817739 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800928    4944 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800932    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800937    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800942    4944 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800946    4944 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800950    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800957    4944 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800961    4944 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800966    4944 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800970    4944 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800974    4944 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800978    4944 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800983    4944 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800987    4944 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800992    4944 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.800996    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801000    4944 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801004    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801008    4944 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 18:45:01.818279 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801015    4944 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801047    4944 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801054    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801060    4944 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801065    4944 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801069    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801073    4944 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801078    4944 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801082    4944 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801087    4944 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801091    4944 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801095    4944 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801100    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801105    4944 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801110    4944 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801116    4944 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801120    4944 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801127    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801132    4944 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 18:45:01.818780 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801137    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801143    4944 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801147    4944 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801151    4944 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801155    4944 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801160    4944 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801164    4944 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801169    4944 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801173    4944 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801177    4944 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801181    4944 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801186    4944 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801190    4944 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801195    4944 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801199    4944 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801203    4944 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801208    4944 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801214    4944 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801220    4944 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801226    4944 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 18:45:01.819416 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801231    4944 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801235    4944 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801239    4944 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801244    4944 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801248    4944 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801252    4944 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801257    4944 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801265    4944 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:01.801271    4944 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.801350    4944 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803417    4944 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803475    4944 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803660    4944 server.go:1014] "Starting client certificate rotation"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803806    4944 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803871    4944 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 18:39:19 +0000 UTC" deadline="2028-08-10 01:39:30.737120154 +0000 UTC"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.803901    4944 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15486h54m28.933221328s"
Nov 03 18:45:01.820208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.804119    4944 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:45:01.820728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.805305    4944 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 18:45:01.820728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.807031    4944 log.go:25] "Validated CRI v1 runtime API"
Nov 03 18:45:01.820728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.808713    4944 log.go:25] "Validated CRI v1 image API"
Nov 03 18:45:01.820728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.809403    4944 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 18:45:01.820728 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.811170    4944 fs.go:135] Filesystem UUIDs: map[2004e218-964e-4db5-b328-8251b3504349:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8a1b3ee9-8eea-4034-98d9-60c9c51d0892:/dev/nvme0n1p4]
Nov 03 18:45:01.820915 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.811191    4944 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ddfbb9d22ccc0c4836927dd7484f7e97cd311227681c35c98adb2404bd65a22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ddfbb9d22ccc0c4836927dd7484f7e97cd311227681c35c98adb2404bd65a22/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17ce785e7c4d51b6d6ae3b0e2b5834ea7f1d0da8dff8cd9ac0d3982ac712de56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17ce785e7c4d51b6d6ae3b0e2b5834ea7f1d0da8dff8cd9ac0d3982ac712de56/userdata/shm major:0 minor:328 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~projected/kube-api-access-ccslj:{mountpoint:/var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~projected/kube-api-access-ccslj major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f511080-dd62-473c-b8d6-14ff2c0aa23d/volumes/kubernetes.io~projected/kube-api-access-cdwrd:{mountpoint:/var/lib/kubelet/pods/2f511080-dd62-473c-b8d6-14ff2c0aa23d/volumes/kubernetes.io~projected/kube-api-access-cdwrd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513d0420-ea97-401f-ad3a-9ee14c44a819/volumes/kubernetes.io~projected/kube-api-access-7sn22:{mountpoint:/var/lib/kubelet/pods/513d0420-ea97-401f-ad3a-9ee14c44a819/volumes/kubernetes.io~projected/kube-api-access-7sn22 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~projected/kube-api-access-n94fx:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~projected/kube-api-access-n94fx major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77675da6-c438-43cd-aa5a-64bc029c93c3/volumes/kubernetes.io~projected/kube-api-access-kmmxk:{mountpoint:/var/lib/kubelet/pods/77675da6-c438-43cd-aa5a-64bc029c93c3/volumes/kubernetes.io~projected/kube-api-access-kmmxk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/777bcab9-01dd-4932-8c5d-0f623570e177/volumes/kubernetes.io~projected/kube-api-access-jr7qb:{mountpoint:/var/lib/kubelet/pods/777bcab9-01dd-4932-8c5d-0f623570e177/volumes/kubernetes.io~projected/kube-api-access-jr7qb major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8326c46d-81bd-4d7f-801b-8a6aa2c9e724/volumes/kubernetes.io~projected/kube-api-access-bxvtp:{mountpoint:/var/lib/kubelet/pods/8326c46d-81bd-4d7f-801b-8a6aa2c9e724/volumes/kubernetes.io~projected/kube-api-access-bxvtp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~projected/kube-api-access-d5fbw:{mountpoint:/var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~projected/kube-api-access-d5fbw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89884da1-8f1b-4d40-a0ca-9d168399a458/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/89884da1-8f1b-4d40-a0ca-9d168399a458/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~projected/kube-api-access-5g6w9:{mountpoint:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~projected/kube-api-access-5g6w9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92594b38-de85-4648-ba98-f27a2da216b9/volumes/kubernetes.io~projected/kube-api-access-hksct:{mountpoint:/var/lib/kubelet/pods/92594b38-de85-4648-ba98-f27a2da216b9/volumes/kubernetes.io~projected/kube-api-access-hksct major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~projected/kube-api-access-b52np:{mountpoint:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~projected/kube-api-access-b52np major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~projected/kube-api-access-b6tv7:{mountpoint:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~projected/kube-api-access-b6tv7 major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~empty-dir/config-out major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/kube-api-access-pnzrg:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/kube-api-access-pnzrg major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/tls-assets major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/config-volume major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/web-config major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bca87fa3-f076-4255-9cb1-6a93dac8ff63/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bca87fa3-f076-4255-9cb1-6a93dac8ff63/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce3d8351-d66e-4646-93e1-d6d5245c5f65/volumes/kubernetes.io~projected/kube-api-access-dc5rd:{mountpoint:/var/lib/kubelet/pods/ce3d8351-d66e-4646-93e1-d6d5245c5f65/volumes/kubernetes.io~projected/kube-api-access-dc5rd major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~projected/kube-api-access-97pnz:{mountpoint:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~projected/kube-api-access-97pnz major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~projected/kube-api-access-8jx9k:{mountpoint:/var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~projected/kube-api-access-8jx9k major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~projected/kube-api-access-xxlht:{mountpoint:/var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~projected/kube-api-access-xxlht major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b151bbd83bd7e623c9301093cf05fcad582fef0597d443169afa6b4b9b6909e3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2bb355bbca6447164f4dbfa06f5baf071e04df24e54d377251a7282e5a5b88ec/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/89ba5cd97bdf9a7261868c8c715fc9ffb52ccca474f2ac615bab4efe671445a3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/12d21ac5fea62a5bf77fa376b7e72eb5810dfff7b430f1eb0aed0adf1259acd0/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ec5cbae76b0048c3e94dc097aed2d3c07793ce098835a90fac5a81074f7104f9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6e63c0b8ff41ef2a1c90a664a62c96c4fa186bb82af8267c1848899fba8ff600/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/db6479e102cc3326136e8b3b1afd10abba71e5a2e606f2c67aa5c70401d0a1b4/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/92e260e52d546737e353329e26a2d35cb05e3bec24d6a8bfeb052068b5af6b1f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/08ac27e91fe7bee31e915a85fe51b9445e3137fbd5173d5aa27b82fe9b40fa51/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5c80a82b7a8f2fd8ec4c7940b90400deed41abc483489f7f2245fcbcd1d5a801/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/a25384d6989d312342b3de5c819e159a23aa52effb7a177b9909538536f87c86/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/9eb761556ba81c2b80b8e416f63aacaa446c41053092af12bf81175d8de0969e/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0ef633cad023b91c19bf4a6f12ff36ea180265e3c7de1d8ac179fd9b688cc214/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/601450db002fdb344c5be40e062aa269163d08fb266d9afcd33981249d90c071/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/65d2c1527827953672b4c996b798cd50e0411ffb7609e9d50c99753dd9eea9c1/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/101a5f36809e76b707e8e3d9cac9cb095bce0d3a7bf70a4c149f597654f5c0cf/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6420b93651db3c4c74f99410961c51d0d82d9a03e8fc3fadf89b508d9d1bddc9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e3a83897078fa86bc412de38ba41652306bc64bc31ccf2217afc84c583888483/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0323edfc215d732df87eae1355c636efe41e8b4d6602f0bc63a9be7ce1508614/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/8a6d48d6a1db9fe5551fe842b0353754aef94e037471fccd71b1d269c756843f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/96b3697fcb09b2e74e3e8348af4f5515c509b3855ba0731f9ef39da36248ac20/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/6dab1ed6e9d7e4f619bec3006074b2515e0d1714711235a1ee1f0a22ff9edf6d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/4e3a485007c4abf733f3ce0e4d9284e33fbd56cb511772b991707a213aa9e5e8/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/7ba6e455f93964ce0966ee0cc3733499fabf8a81794a188d736736126400fa52/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/dd7453b58410bf2b74eda39983e6ed320860f773e93131d0fbebdb4d1fffa970/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/e79e94e9d91490dc1605e9837ebd5c2e6338e04a1de91f36aca8640aa7ba4a89/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/3aabfcbb91ddf992c5bb755908303fc3ee344ebf840ec9d987ce091dc59117bb/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/2babba5530846395128f101d7da60843ef5f6ea000285a029e4c3c7f960e59cb/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/2051fd5ea76b5d8bc34a1ec96fa511312a40b940f98e7a1aa16c6b51bece7b5f/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/72330e5ef181ff68b2890ba119286f573b9424848737072216604cab74048129/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/85a7f9db6a1e296cb2e8ab3763a4ac9ba0d737910b2e62eabc88aaf711f9882e/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/e2a5932a0c18c3d4064afbc51c64986d876445c5c5ea1cc0b291168be63ee1a1/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/75c3ca51b49dae36b10650899092fdb4f2e6b1df0815d01fdbc38b92842a45af/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/20596fd0d2987c7f25b3217199cf9ba781633347889f306f8dc3c4f78a0f7e15/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/f8a939781764c6b60e46693affb1b67b678610f13d034aa845e8b560722280fe/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/2bfa57f4569c9ba4a3903c934ae3a172ca4e0e35bfb373a9426e6acd47b1ee44/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/c2fb4328e266eac954670aa46eafff8354d2dee0decb49dc494dee96fad87caa/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/e4186607e7e9f892ab8ab379bf0e7cf5e9af2985ff9605bc070a0cbe677df6d5/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/29022df62bc9d9ec50dd470fe409d9364ca6739e0959d9172bbd3b93f88cb61f/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/df3a80e4c3800de0587f56ceaf93efeabe7a4e6679bb0ddd41a3e74f5e57a180/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/29bde0d358a467f1f33437389fb99201c87f2a60c38e1b57d70310d4ce48ce36/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/1f5348304aadf464608c24344d4a9044d0e4af1ee7c73ccaf29fb7379238ffc9/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/79e8defec6230d74d44cbc9e673ae4b0196320c095275555899f850940b809a1/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c4d1ff964ef7ecc45a26a8e9b51b1784960214034d3bc7a5573eb4e0ca507235/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/35eee749a4a8e57054fedfc087391951db4dbc0d0f4de010c38836eb86f5f5d1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/509f6ca33ab9e1ddc64cfc58ad3dfda377790312e207f8c20a1c0b079eef8a8e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/aebb324d1044a678c24b8fbe6991aed490e8bc4cfb1856889e604598ee5bc33a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fb39364e276d43bfee20684848e585c34920721ba66dbe16f3606606c790eff8/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1d055085ab9b5f78ec260fff51559ded40b62050aeb2aac60b3ca1a962bde7a7/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a503125b8e5776421b127f189a80926b93dfc2ce36e42ee3c1a6ee312b762757/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/40fe26ca495b793b97149600a2213a7be29d18e3038a0b0302ffe043d302ebb7/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.818673    4944 manager.go:217] Machine: {Timestamp:2026-11-03 18:45:01.818214417 +0000 UTC m=+0.095567992 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105488 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f08ce56531714359b90e317507f5f SystemUUID:ec2f08ce-5653-1714-359b-90e317507f5f BootID:576bcac4-ade8-40a6-8302-e8a4142c8c94 Filesystems:[{Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f511080-dd62-473c-b8d6-14ff2c0aa23d/volumes/kubernetes.io~projected/kube-api-access-cdwrd DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8565a95832115027653ef3c1c6fa359e049d1fda95f5b7a29029453468b70565/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~projected/kube-api-access-ccslj DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c5209060a89ac763211edfdbfa037efc23dd73818fc10317a2e8ed7c7f675964/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5737bd70f55b1cb52e601e35910916f5244023e78babfeb12d2d3fa98f1260a4/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77675da6-c438-43cd-aa5a-64bc029c93c3/volumes/kubernetes.io~projected/kube-api-access-kmmxk DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~projected/kube-api-access-d5fbw DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/89884da1-8f1b-4d40-a0ca-9d168399a458/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a5aa2620baeef57fcb7f1d3b3bb6a170fe9e9214a9e24264706a24d537c562a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~projected/kube-api-access-n94fx DeviceMajor:0 DeviceMinor:325 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volumes/kubernetes.io~projected/kube-api-access-5g6w9 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca40aae7aae02fa0f43b02530587581325c00548f29a789579d8f41e45f64784/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bca87fa3-f076-4255-9cb1-6a93dac8ff63/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:222 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:324 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~projected/kube-api-access-8jx9k DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:313 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~projected/kube-api-access-97pnz DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d898b8d60c35f98a8039c397db1400c40e07b6b7c34a50fc07e29a4026f1ed20/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dfb8edf817d3e4c59ce6b588d41d4a98cd3a0bfd73abf8cd393dc0e16fb5ce73/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce3d8351-d66e-4646-93e1-d6d5245c5f65/volumes/kubernetes.io~projected/kube-api-access-dc5rd DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~projected/kube-api-access-b6tv7 DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b379ed46037970f3c14fa8191455e34de649ff3b9b0269630c8a99d011900179/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffd60273ae1dc11950d0d8d6dbc28b774cd0d2af9d36302548c3bd077eec065d/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17ce785e7c4d51b6d6ae3b0e2b5834ea7f1d0da8dff8cd9ac0d3982ac712de56/userdata/shm DeviceMajor:0 DeviceMinor:328 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e20f1d5f-64dc-42af-bed3-167aab7074ee/volumes/kubernetes.io~projected/kube-api-access-xxlht DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67a5bd0de009524c490cf7a5e71e7bf73d2d747254ce5df308c94577a0e3b6d9/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:323 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d5c700ed-1257-476f-a0e8-2a7483d06af7/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ddfbb9d22ccc0c4836927dd7484f7e97cd311227681c35c98adb2404bd65a22/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/513d0420-ea97-401f-ad3a-9ee14c44a819/volumes/kubernetes.io~projected/kube-api-access-7sn22 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/221317e578f55c13bd5048e5c3ead466c65576086f8088c40e121a64ac1cab09/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c7615f635196e757b14543f6b1527e2f5c318a28f92629dc6226a52450a9199/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/777bcab9-01dd-4932-8c5d-0f623570e177/volumes/kubernetes.io~projected/kube-api-access-jr7qb DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:280 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~projected/kube-api-access-pnzrg DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10e763e22e44fe217a38c2cf79e10b4e3bb51e6b87f3c87b77c499a6637e7616/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0/volumes/kubernetes.io~projected/kube-api-access-b52np DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/86362a0d-cbf8-4478-b304-f6de2fd0bb0f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/618c170c-3333-4548-864b-2e1d78308333/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8326c46d-81bd-4d7f-801b-8a6aa2c9e724/volumes/kubernetes.io~projected/kube-api-access-bxvtp DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92594b38-de85-4648-ba98-f27a2da216b9/volumes/kubernetes.io~projected/kube-api-access-hksct DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07e433c68b16e5fd82d555cfed993671b11c9c5ecd59c6eff4204effd0c3f4d9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08a6bf43-66d1-4875-9821-58d2a9ff008d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9c31aa03976675857c3fe31a825bf8ea01b66b7d48d13e4df18f8d1ca0ef87c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce552fcf-36e3-435c-887d-6ceff825d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:270 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1566386964dbde59c07c549f37b86743e029affe0d72737f3cc8157bcdd2a527/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ddfbb9d22ccc0c MacAddress:6a:60:58:af:da:bf Speed:10000 Mtu:8901} {Name:17ce785e7c4d51b MacAddress:1a:04:2b:ec:74:c8 Speed:10000 Mtu:8901} {Name:3c7615f635196e7 MacAddress:26:ef:08:b4:29:be Speed:10000 Mtu:8901} {Name:5737bd70f55b1cb MacAddress:22:c9:cc:12:93:c7 Speed:10000 Mtu:8901} {Name:67a5bd0de009524 MacAddress:6a:96:29:b4:22:cb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:a8:9b:1f Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:00:30:79:d5:d2 Speed:0 Mtu:8901} {Name:c5209060a89ac76 MacAddress:02:cd:18:6d:c5:07 Speed:10000 Mtu:8901} {Name:ca40aae7aae02fa MacAddress:2a:04:c0:5d:d0:83 Speed:10000 Mtu:8901} {Name:d898b8d60c35f98 MacAddress:56:42:39:8d:fc:a3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:a8:9b:1f Speed:0 Mtu:9001} {Name:ffd60273ae1dc11 MacAddress:ce:9f:b7:5b:1f:6b Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:e0:b7:a8:fd:2e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:df:56:10:29:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.818994    4944 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819039    4944 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819535    4944 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819566    4944 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-192.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819925    4944 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819939    4944 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819954    4944 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.819972    4944 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820137    4944 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820442    4944 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820823    4944 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820862    4944 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820881    4944 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820896    4944 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820915    4944 kubelet.go:405] "Adding apiserver pod source"
Nov 03 18:45:01.821294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.820946    4944 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 18:45:01.822265 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.821756    4944 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 18:45:01.822265 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.821767    4944 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:45:01.822265 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.822240    4944 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 18:45:01.886847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.886814    4944 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 18:45:01.886847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.886834    4944 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 18:45:01.887007 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.886986    4944 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 18:45:01.887244 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887227    4944 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 18:45:01.887313 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887268    4944 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 18:45:01.887409 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887397    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887419    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887430    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887439    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887449    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887458    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 18:45:01.887468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887468    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887477    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887488    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887498    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887519    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887534    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887575    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 18:45:01.887736 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.887586    4944 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 18:45:01.891108 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.891090    4944 server.go:1263] "Started kubelet"
Nov 03 18:45:01.891231 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.891168    4944 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 18:45:01.891277 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.891209    4944 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 18:45:01.891317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.891288    4944 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 18:45:01.891688 ip-10-0-137-192 systemd[1]: Started Kubernetes Kubelet.
Nov 03 18:45:01.892574 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.891716    4944 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 18:45:01.892684 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.892651    4944 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 18:45:01.896186 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896007    4944 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 18:45:01.896284 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:01.896218    4944 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:45:01.896405 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896369    4944 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 18:45:01.896405 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896389    4944 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 18:45:01.896531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896404    4944 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 18:45:01.896531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896440    4944 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 18:39:21 +0000 UTC" deadline="2028-06-06 01:08:51.273862604 +0000 UTC"
Nov 03 18:45:01.896531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896475    4944 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 18:45:01.896531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.896474    4944 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13926h23m49.377391829s"
Nov 03 18:45:01.898009 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.897986    4944 factory.go:55] Registering systemd factory
Nov 03 18:45:01.898009 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.898016    4944 factory.go:223] Registration of the systemd container factory successfully
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900266    4944 factory.go:153] Registering CRI-O factory
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900287    4944 factory.go:223] Registration of the crio container factory successfully
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900354    4944 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900375    4944 factory.go:103] Registering Raw factory
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900394    4944 manager.go:1199] Started watching for new ooms in manager
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900760    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" volumeName="kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900812    4944 manager.go:319] Starting recovery of all containers
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900816    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900835    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900846    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900855    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900866    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86362a0d-cbf8-4478-b304-f6de2fd0bb0f" volumeName="kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900879    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900890    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca" seLinuxMountContext=""
Nov 03 18:45:01.900926 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900902    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900949    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="86362a0d-cbf8-4478-b304-f6de2fd0bb0f" volumeName="kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.900971    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92594b38-de85-4648-ba98-f27a2da216b9" volumeName="kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901002    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901019    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901036    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce552fcf-36e3-435c-887d-6ceff825d956" volumeName="kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901060    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901076    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce552fcf-36e3-435c-887d-6ceff825d956" volumeName="kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901088    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901099    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901108    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce3d8351-d66e-4646-93e1-d6d5245c5f65" volumeName="kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901120    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce552fcf-36e3-435c-887d-6ceff825d956" volumeName="kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901135    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c700ed-1257-476f-a0e8-2a7483d06af7" volumeName="kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901149    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" volumeName="kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901159    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f511080-dd62-473c-b8d6-14ff2c0aa23d" volumeName="kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901168    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901179    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901194    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901208    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901224    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bca87fa3-f076-4255-9cb1-6a93dac8ff63" volumeName="kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901257    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c700ed-1257-476f-a0e8-2a7483d06af7" volumeName="kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901273    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901288    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901316    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce552fcf-36e3-435c-887d-6ceff825d956" volumeName="kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901335    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a6bf43-66d1-4875-9821-58d2a9ff008d" volumeName="kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901350    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513d0420-ea97-401f-ad3a-9ee14c44a819" volumeName="kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901360    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89884da1-8f1b-4d40-a0ca-9d168399a458" volumeName="kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901369    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" volumeName="kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901379    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92594b38-de85-4648-ba98-f27a2da216b9" volumeName="kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901392    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92594b38-de85-4648-ba98-f27a2da216b9" volumeName="kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901409    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901425    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513d0420-ea97-401f-ad3a-9ee14c44a819" volumeName="kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901439    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901453    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="777bcab9-01dd-4932-8c5d-0f623570e177" volumeName="kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901469    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89884da1-8f1b-4d40-a0ca-9d168399a458" volumeName="kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901480    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901492    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e20f1d5f-64dc-42af-bed3-167aab7074ee" volumeName="kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901504    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f511080-dd62-473c-b8d6-14ff2c0aa23d" volumeName="kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901512    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92594b38-de85-4648-ba98-f27a2da216b9" volumeName="kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901520    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0" volumeName="kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901539    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0" volumeName="kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901552    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901564    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e20f1d5f-64dc-42af-bed3-167aab7074ee" volumeName="kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901575    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901585    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901596    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0" volumeName="kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901607    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b06794b8-5669-4fda-901e-40b6209d79b4" volumeName="kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901618    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c700ed-1257-476f-a0e8-2a7483d06af7" volumeName="kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901636    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" volumeName="kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901645    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8326c46d-81bd-4d7f-801b-8a6aa2c9e724" volumeName="kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp" seLinuxMountContext=""
Nov 03 18:45:01.901618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901653    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" volumeName="kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901661    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce3d8351-d66e-4646-93e1-d6d5245c5f65" volumeName="kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901670    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e20f1d5f-64dc-42af-bed3-167aab7074ee" volumeName="kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901679    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08a6bf43-66d1-4875-9821-58d2a9ff008d" volumeName="kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901687    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901694    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77675da6-c438-43cd-aa5a-64bc029c93c3" volumeName="kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901702    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" volumeName="kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901710    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77675da6-c438-43cd-aa5a-64bc029c93c3" volumeName="kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901729    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5c700ed-1257-476f-a0e8-2a7483d06af7" volumeName="kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901737    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e20f1d5f-64dc-42af-bed3-167aab7074ee" volumeName="kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901746    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="618c170c-3333-4548-864b-2e1d78308333" volumeName="kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901753    4944 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77675da6-c438-43cd-aa5a-64bc029c93c3" volumeName="kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy" seLinuxMountContext=""
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901761    4944 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 18:45:01.903894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.901767    4944 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 18:45:01.917666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.916195    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:45:01.917666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.917052    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:45:01.924800 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.924738    4944 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 18:45:01.926218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.926195    4944 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 18:45:01.926218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.926217    4944 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 18:45:01.926370 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.926236    4944 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 18:45:01.926370 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:01.926275    4944 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 18:45:01.942297 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.942264    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:45:01.942642 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.942600    4944 generic.go:356] "Generic (PLEG): container finished" podID="8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0" containerID="294ffb7ce89b4dfe72acdb7b2e5d709c62d5e0e350f6e5594db77a1380f7b859" exitCode=1
Nov 03 18:45:01.957169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957123    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="e0e68128b62d6e482260a26b858e3dc846764de23e30f99d8f52301e65fda4a4" exitCode=0
Nov 03 18:45:01.957169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957157    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="1d74cdf8ba23ff5e53870c6888ce96158b9b67127ce685684b362e9235cf0990" exitCode=0
Nov 03 18:45:01.957169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957164    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="e5e15f3ac717adf819a6204cf3c6d524e0d0d29fb2c9b2de9082f1ae06b72ed6" exitCode=0
Nov 03 18:45:01.957169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957169    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="daca2c8bc9925f1f134e3877336867c96dfdcbace156bad5b30d5f7c8c896013" exitCode=0
Nov 03 18:45:01.957169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957175    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="f7a3f32de0a43a89b39a6116637bef818d83da71face623b8401c14f157fac6a" exitCode=0
Nov 03 18:45:01.957471 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.957183    4944 generic.go:356] "Generic (PLEG): container finished" podID="92594b38-de85-4648-ba98-f27a2da216b9" containerID="dbcc31f7222a228bdf84b9894b4004df82ab08602c4a4d1bfe4dba7e7ffd7582" exitCode=0
Nov 03 18:45:01.959164 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.959136    4944 generic.go:356] "Generic (PLEG): container finished" podID="8bcbaad710a39449db7d66d55191eacf" containerID="226db2a07d8acd618382bf4337dc41ed7284f5535114067958a58fe85f483606" exitCode=0
Nov 03 18:45:01.962561 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.962532    4944 generic.go:356] "Generic (PLEG): container finished" podID="a592af2a-0e6b-4455-b71f-bf6c8fe5a14c" containerID="ebacf2b5164b80ee4e2aa9c154cdc65055be6c7fb59ad6c70fb1d1f801db9781" exitCode=0
Nov 03 18:45:01.981338 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.981318    4944 manager.go:324] Recovery completed
Nov 03 18:45:01.996868 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:01.996844    4944 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:45:01.998623 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:01.998609    4944 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:45:02.000079 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.000058    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:45:02.000180 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.000089    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:45:02.000180 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.000099    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:45:02.002616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002597    4944 cpu_manager.go:226] "Starting" policy="none"
Nov 03 18:45:02.002616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002608    4944 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:45:02.002754 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002611    4944 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 18:45:02.002754 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002679    4944 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 18:45:02.002839 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002817    4944 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002828    4944 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002845    4944 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002850    4944 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002855    4944 policy_none.go:50] "Start"
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002866    4944 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 18:45:02.002886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002875    4944 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 18:45:02.003142 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002992    4944 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 18:45:02.003142 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.002999    4944 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 18:45:02.003142 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.003004    4944 policy_none.go:44] "Start"
Nov 03 18:45:02.003515 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.003501    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:45:02.003576 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.003527    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:45:02.003576 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.003539    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:45:02.007727 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.007644    4944 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 18:45:02.011981 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.011957    4944 manager.go:341] "Starting Device Plugin manager"
Nov 03 18:45:02.012090 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.011983    4944 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 18:45:02.012090 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.011993    4944 server.go:87] "Starting device plugin registration server"
Nov 03 18:45:02.012321 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012308    4944 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 18:45:02.012386 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012325    4944 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 18:45:02.012432 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012396    4944 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 18:45:02.012536 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012493    4944 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 18:45:02.012536 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012533    4944 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 18:45:02.012536 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012540    4944 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 18:45:02.013198 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.012575    4944 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T18:45:02.012523167Z","UUID":"8284e626-cb9e-4141-a7ad-96e193835644","Handler":null,"Name":"","Endpoint":""}
Nov 03 18:45:02.014237 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.014214    4944 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 18:45:02.014331 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.014261    4944 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-192.us-west-2.compute.internal\" not found"
Nov 03 18:45:02.014331 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.014309    4944 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 18:45:02.014406 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.014336    4944 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 18:45:02.019047 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.019030    4944 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 18:45:02.019714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.019697    4944 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:45:02.019808 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.019769    4944 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:45:02.027275 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027238    4944 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"]
Nov 03 18:45:02.027811 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027765    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerStarted","Data":"a2357aa3362e69d15abc9d68fbaef8e155d844da2573f2a4755a43424eb63770"}
Nov 03 18:45:02.027900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027813    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerDied","Data":"226db2a07d8acd618382bf4337dc41ed7284f5535114067958a58fe85f483606"}
Nov 03 18:45:02.027900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027827    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"8bcbaad710a39449db7d66d55191eacf","Type":"ContainerStarted","Data":"6ad3cc260541b56e165bd136d5b5ed39c83a374cfc471ed2dfb2e9b26c71da98"}
Nov 03 18:45:02.027900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027839    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"5b200ea252cb76e83bc321c9ddace574","Type":"ContainerStarted","Data":"872561eceeb967a7e743daff53fb45849d81d85313113e625deeaee607764c16"}
Nov 03 18:45:02.027900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027850    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal" event={"ID":"5b200ea252cb76e83bc321c9ddace574","Type":"ContainerStarted","Data":"7a48fef51da05a782a01ccc7963b2dfb401670792e37f126c02c4fd2627e7853"}
Nov 03 18:45:02.027900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027850    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.028088 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.027954    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.066760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.066727    4944 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:45:02.097065 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.097034    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.111712 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.111685    4944 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 18:45:02.111898 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.111879    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:02.111946 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.111882    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:02.111984 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.111941    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.111984 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.111968    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.112799 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.112776    4944 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 18:45:02.113722 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.113710    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 18:45:02.113804 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.113735    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 18:45:02.113804 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.113745    4944 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-192.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 18:45:02.113804 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.113782    4944 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.120014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.119998    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.120066 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.120025    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.120066 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.120043    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.154220 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.154194    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:02.154285 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.154241    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.154285 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.154254    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221092 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221022    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221092 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221055    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221092 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221076    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221254 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221135    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221254 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221192    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5b200ea252cb76e83bc321c9ddace574-config\") pod \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"5b200ea252cb76e83bc321c9ddace574\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.221254 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.221198    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcbaad710a39449db7d66d55191eacf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" (UID: \"8bcbaad710a39449db7d66d55191eacf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.225277 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.225261    4944 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.225336 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.225325    4944 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.225713 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.225690    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:02.225805 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:02.225728    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.821741 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.821710    4944 apiserver.go:52] "Watching apiserver"
Nov 03 18:45:02.974185 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.974114    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.974365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.974305    4944 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:02.974429 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.974412    4944 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 18:45:02.975149 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.975119    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-4sblg","openshift-ovn-kubernetes/ovnkube-node-5g7pb","openshift-dns/node-resolver-mfgjk","openshift-monitoring/node-exporter-gd5jv","kube-system/konnectivity-agent-ddzdz","openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v","openshift-monitoring/thanos-querier-74994bbc58-d8mv2","openshift-dns/dns-default-mq4br","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-4xkkk","openshift-ingress-canary/ingress-canary-qvss5","openshift-network-diagnostics/network-check-target-7dp7q","openshift-network-operator/iptables-alerter-tddmx","kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-q52x4","kube-system/global-pull-secret-syncer-94hv8","openshift-cluster-node-tuning-operator/tuned-kx25z","openshift-insights/insights-runtime-extractor-l8qkk","openshift-multus/multus-jv87r"]
Nov 03 18:45:02.985888 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.985865    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:02.986535 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.986517    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:02.998413 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:02.998386    4944 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 18:45:03.025454 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025421    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.025454 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025435    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025471    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025508    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025522    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025509    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvtp\" (UniqueName: \"kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025577    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025591    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025601    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.025689 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025658    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccslj\" (UniqueName: \"kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025729    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025757    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025783    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025828    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025851    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025876    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-textfile\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025904    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025931    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmxk\" (UniqueName: \"kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.025981    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026006    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:45:03.026097 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026029    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026153    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026182    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026249    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026282    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-cni-binary-copy\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026295    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026330    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026356    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026457    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026489    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026517    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026553    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.026625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026596    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5rd\" (UniqueName: \"kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026649    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026669    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jx9k\" (UniqueName: \"kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026688    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026715    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026738    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026777    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e20f1d5f-64dc-42af-bed3-167aab7074ee-data-volume\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026784    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026837    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026886    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026915    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026936    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026952    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026969    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6w9\" (UniqueName: \"kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026984    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdwrd\" (UniqueName: \"kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.026998    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027013    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027029    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027038    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce3d8351-d66e-4646-93e1-d6d5245c5f65-tmp-dir\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027051    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n94fx\" (UniqueName: \"kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027084    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027112    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027137    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027168    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.027201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027197    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027223    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6tv7\" (UniqueName: \"kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027249    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027280    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027304    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027333    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027385    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027414    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pnz\" (UniqueName: \"kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027438    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027469    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027498    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027526    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027551    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027575    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027605    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027633    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027695    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027716    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027735    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027755    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027767    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-tuned\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027773    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027845    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027863    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027883    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027903    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027918    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027963    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.027981    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028016    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028033    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028106    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028128    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028163    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028181    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52np\" (UniqueName: \"kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028204    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028222    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028237    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028253    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028161    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-tmp\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.028345 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028327    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028412    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5c700ed-1257-476f-a0e8-2a7483d06af7-tmp-dir\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028464    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028481    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028522    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028554    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlht\" (UniqueName: \"kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028598    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028661    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028665    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/618c170c-3333-4548-864b-2e1d78308333-tmp\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028698    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028745    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028778    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028834    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028900    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.028999    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sn22\" (UniqueName: \"kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029045    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029078    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029129    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029166    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029220    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029268    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029302    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029319    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029374    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029423    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029501    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029534    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029568    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029614    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029718    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fbw\" (UniqueName: \"kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029763    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029863    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-binary-copy\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.029935    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030024    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.030063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030064    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hksct\" (UniqueName: \"kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030090    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030117    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030144    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030170    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030197    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030239    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030302    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030329    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030353    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030388    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030415    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.031246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.030435    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.032432 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032407    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 18:45:03.032432 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032423    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.032622 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032463    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 18:45:03.032622 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032526    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 18:45:03.032622 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032544    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:03.032622 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032578    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.032622 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:03.032600    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:03.032872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032815    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.032872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032830    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 18:45:03.032872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032833    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.032872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032871    4944 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 18:45:03.033016 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032875    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.033016 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.032915    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.033016 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:03.032920    4944 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-192.us-west-2.compute.internal"
Nov 03 18:45:03.033143 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.033037    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 18:45:03.033143 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.033108    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.033212 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.033169    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 18:45:03.036361 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.036342    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-env-overrides\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.036462 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.036343    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/89884da1-8f1b-4d40-a0ca-9d168399a458-konnectivity-ca\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:45:03.036462 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.036446    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/513d0420-ea97-401f-ad3a-9ee14c44a819-iptables-alerter-script\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.038564 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.038547    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-daemon-config\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.039739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.039720    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-config\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.040916 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.040899    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovnkube-script-lib\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.058651 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058632    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.058771 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058753    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 18:45:03.058848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058755    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 18:45:03.058848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058778    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.058848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058780    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.058848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058800    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.058991 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058864    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.058991 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.058914    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 18:45:03.068582 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.068557    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.068766 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.068751    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92594b38-de85-4648-ba98-f27a2da216b9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.068856 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.068808    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.068897 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.068857    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 18:45:03.069011 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.068995    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2f511080-dd62-473c-b8d6-14ff2c0aa23d-serviceca\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.069760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.069748    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.070210 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070185    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 18:45:03.070322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070240    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.070481 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070464    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 18:45:03.070481 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070477    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.070606 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070528    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 18:45:03.070606 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070555    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 18:45:03.070719 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.070634    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 18:45:03.071658 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.071643    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 18:45:03.076496 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.076478    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.078552 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.078534    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.078835 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.078817    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-accelerators-collector-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.079342 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.079322    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce552fcf-36e3-435c-887d-6ceff825d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.079420 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.079384    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c700ed-1257-476f-a0e8-2a7483d06af7-config-volume\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.079516 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.079501    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.079841 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.079820    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-metrics-client-ca\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.079976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.079960    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-97rln\""
Nov 03 18:45:03.080050 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080025    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 18:45:03.080105 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080069    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 18:45:03.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080118    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 18:45:03.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080125    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 18:45:03.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080121    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 18:45:03.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080144    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 18:45:03.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080150    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080152    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080185    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080190    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080165    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qnvrx\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080223    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080248    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080226    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 18:45:03.080378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080309    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/618c170c-3333-4548-864b-2e1d78308333-metrics-client-ca\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080492    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080511    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d8dtx\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080551    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnhrd\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080114    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080595    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-45nsg\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080623    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-99m4h\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080627    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jwgdr\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080649    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-60agudo7g89ai\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080657    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-q46mt\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080684    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080692    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8n5dg\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080700    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gq4wc\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080702    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ltsf8\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080707    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4xp5l\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080637    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-grblk\""
Nov 03 18:45:03.080738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080736    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080766    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080772    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080781    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sjcxp\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080807    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080822    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 18:45:03.081311 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.080855    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081316    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081343    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7r6fd\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081318    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081348    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bctnd\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081521    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4ltsw\""
Nov 03 18:45:03.081580 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081548    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jlxrp\""
Nov 03 18:45:03.081848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081720    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-k4rz5\""
Nov 03 18:45:03.081848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.081782    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 18:45:03.086439 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.086420    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-tls\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.086544 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.086460    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/89884da1-8f1b-4d40-a0ca-9d168399a458-agent-certs\") pod \"konnectivity-agent-ddzdz\" (UID: \"89884da1-8f1b-4d40-a0ca-9d168399a458\") " pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:45:03.086544 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.086511    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bca87fa3-f076-4255-9cb1-6a93dac8ff63-original-pull-secret\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.086638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.086586    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.086638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.086585    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.087154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.087135    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d5c700ed-1257-476f-a0e8-2a7483d06af7-metrics-tls\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.087187 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.087150    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088259 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088240    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088335 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088242    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.088335 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088268    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088335 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088272    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088335 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088247    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088335 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088320    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08a6bf43-66d1-4875-9821-58d2a9ff008d-cert\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:45:03.088508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088362    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088362    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.088508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088478    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.088508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088488    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.088617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088534    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e20f1d5f-64dc-42af-bed3-167aab7074ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.088617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.088605    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.089555 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.089540    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.089639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.089587    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.089639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.089608    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ce552fcf-36e3-435c-887d-6ceff825d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.090733 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.090719    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.090772 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.090759    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/618c170c-3333-4548-864b-2e1d78308333-secret-grpc-tls\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.090823 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.090767    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-metrics-certs\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5rd\" (UniqueName: \"kubernetes.io/projected/ce3d8351-d66e-4646-93e1-d6d5245c5f65-kube-api-access-dc5rd\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106116    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdwrd\" (UniqueName: \"kubernetes.io/projected/2f511080-dd62-473c-b8d6-14ff2c0aa23d-kube-api-access-cdwrd\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106166    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"alertmanager-main-0\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106261    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmxk\" (UniqueName: \"kubernetes.io/projected/77675da6-c438-43cd-aa5a-64bc029c93c3-kube-api-access-kmmxk\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106330    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6tv7\" (UniqueName: \"kubernetes.io/projected/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-kube-api-access-b6tv7\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106357    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccslj\" (UniqueName: \"kubernetes.io/projected/08a6bf43-66d1-4875-9821-58d2a9ff008d-kube-api-access-ccslj\") pod \"ingress-canary-qvss5\" (UID: \"08a6bf43-66d1-4875-9821-58d2a9ff008d\") " pod="openshift-ingress-canary/ingress-canary-qvss5"
Nov 03 18:45:03.106415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106406    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n94fx\" (UniqueName: \"kubernetes.io/projected/618c170c-3333-4548-864b-2e1d78308333-kube-api-access-n94fx\") pod \"thanos-querier-74994bbc58-d8mv2\" (UID: \"618c170c-3333-4548-864b-2e1d78308333\") " pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.106882 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106645    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sn22\" (UniqueName: \"kubernetes.io/projected/513d0420-ea97-401f-ad3a-9ee14c44a819-kube-api-access-7sn22\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.106882 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106751    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52np\" (UniqueName: \"kubernetes.io/projected/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-kube-api-access-b52np\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.106882 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106877    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr7qb\" (UniqueName: \"kubernetes.io/projected/777bcab9-01dd-4932-8c5d-0f623570e177-kube-api-access-jr7qb\") pod \"network-check-target-7dp7q\" (UID: \"777bcab9-01dd-4932-8c5d-0f623570e177\") " pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:45:03.107006 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106895    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fbw\" (UniqueName: \"kubernetes.io/projected/86362a0d-cbf8-4478-b304-f6de2fd0bb0f-kube-api-access-d5fbw\") pod \"network-metrics-daemon-4sblg\" (UID: \"86362a0d-cbf8-4478-b304-f6de2fd0bb0f\") " pod="openshift-multus/network-metrics-daemon-4sblg"
Nov 03 18:45:03.107006 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.106905    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6w9\" (UniqueName: \"kubernetes.io/projected/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-kube-api-access-5g6w9\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.107062 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.107036    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvtp\" (UniqueName: \"kubernetes.io/projected/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kube-api-access-bxvtp\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.107151 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.107129    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pnz\" (UniqueName: \"kubernetes.io/projected/ce552fcf-36e3-435c-887d-6ceff825d956-kube-api-access-97pnz\") pod \"openshift-state-metrics-7bd7489957-5q2l5\" (UID: \"ce552fcf-36e3-435c-887d-6ceff825d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5"
Nov 03 18:45:03.107186 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.107132    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jx9k\" (UniqueName: \"kubernetes.io/projected/d5c700ed-1257-476f-a0e8-2a7483d06af7-kube-api-access-8jx9k\") pod \"dns-default-mq4br\" (UID: \"d5c700ed-1257-476f-a0e8-2a7483d06af7\") " pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:03.115895 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.115875    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlht\" (UniqueName: \"kubernetes.io/projected/e20f1d5f-64dc-42af-bed3-167aab7074ee-kube-api-access-xxlht\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.115959 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.115910    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hksct\" (UniqueName: \"kubernetes.io/projected/92594b38-de85-4648-ba98-f27a2da216b9-kube-api-access-hksct\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.130892 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130868    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.130958 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130897    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.130958 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130912    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.130958 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130929    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.130958 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130944    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130958    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130972    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130991    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130998    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-registration-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131007    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.130998    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131026    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-hostroot\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131038    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-node-exporter-wtmp\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131015    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131072    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-log-socket\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-multus-certs\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-var-lib-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131082    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131032    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-conf-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131116    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131135    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-socket-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131140    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-cnibin\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.131154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131143    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131179    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131183    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-kubernetes\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131204    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131227    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131251    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131261    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-etc-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131277    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131287    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131301    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131280    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-systemd\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131325    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131346    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-slash\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131352    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131361    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-multus\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131374    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-lib-modules\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131375    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131388    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-netns\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131381    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-ovn\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131406    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-var-lib-kubelet\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131410    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131435    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131459    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131467    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysconfig\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131460    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-kubelet\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131483    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131497    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-netns\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131504    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131518    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-var-lib-cni-bin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131541    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131562    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131580    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131594    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131549    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-run\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131621    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-cnibin\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131629    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-bin\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131624    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131642    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-sys\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131654    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92594b38-de85-4648-ba98-f27a2da216b9-system-cni-dir\") pod \"multus-additional-cni-plugins-q52x4\" (UID: \"92594b38-de85-4648-ba98-f27a2da216b9\") " pod="openshift-multus/multus-additional-cni-plugins-q52x4"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131659    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-sys-fs\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131671    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131655    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-systemd-units\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131702    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131730    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131749    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131760    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131781    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131801    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131824    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2f511080-dd62-473c-b8d6-14ff2c0aa23d-host\") pod \"node-ca-4xkkk\" (UID: \"2f511080-dd62-473c-b8d6-14ff2c0aa23d\") " pod="openshift-image-registry/node-ca-4xkkk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131832    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce3d8351-d66e-4646-93e1-d6d5245c5f65-hosts-file\") pod \"node-resolver-mfgjk\" (UID: \"ce3d8351-d66e-4646-93e1-d6d5245c5f65\") " pod="openshift-dns/node-resolver-mfgjk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131825    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131868    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131900    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131928    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e20f1d5f-64dc-42af-bed3-167aab7074ee-crio-socket\") pod \"insights-runtime-extractor-l8qkk\" (UID: \"e20f1d5f-64dc-42af-bed3-167aab7074ee\") " pod="openshift-insights/insights-runtime-extractor-l8qkk"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131934    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-host-run-k8s-cni-cncf-io\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131947    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-kubelet\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131959    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131975    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.131992    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.131993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132011    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132034    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132053    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-systemd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132059    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132061    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-multus-socket-dir-parent\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132095    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132103    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-dbus\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132097    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-etc-kubernetes\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132124    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-sys\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132124    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132136    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132136    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-modprobe-d\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132146    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-host-cni-netd\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132154    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132175    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132189    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132203    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132218    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77675da6-c438-43cd-aa5a-64bc029c93c3-system-cni-dir\") pod \"multus-jv87r\" (UID: \"77675da6-c438-43cd-aa5a-64bc029c93c3\") " pod="openshift-multus/multus-jv87r"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132224    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132230    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-device-dir\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132247    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132272    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132278    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8326c46d-81bd-4d7f-801b-8a6aa2c9e724-etc-selinux\") pod \"aws-ebs-csi-driver-node-4966v\" (UID: \"8326c46d-81bd-4d7f-801b-8a6aa2c9e724\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4966v"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132285    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bca87fa3-f076-4255-9cb1-6a93dac8ff63-kubelet-config\") pod \"global-pull-secret-syncer-94hv8\" (UID: \"bca87fa3-f076-4255-9cb1-6a93dac8ff63\") " pod="kube-system/global-pull-secret-syncer-94hv8"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132249    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-run-openvswitch\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132294    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132324    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132329    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/513d0420-ea97-401f-ad3a-9ee14c44a819-host-slash\") pod \"iptables-alerter-tddmx\" (UID: \"513d0420-ea97-401f-ad3a-9ee14c44a819\") " pod="openshift-network-operator/iptables-alerter-tddmx"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132351    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132387    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-host\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132388    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0-node-log\") pod \"ovnkube-node-5g7pb\" (UID: \"8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132408    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a592af2a-0e6b-4455-b71f-bf6c8fe5a14c-root\") pod \"node-exporter-gd5jv\" (UID: \"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c\") " pod="openshift-monitoring/node-exporter-gd5jv"
Nov 03 18:45:03.133616 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.132459    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0-etc-sysctl-conf\") pod \"tuned-kx25z\" (UID: \"94db9f0f-6fb5-4aea-bbf7-0c3afa685fb0\") " pod="openshift-cluster-node-tuning-operator/tuned-kx25z"
Nov 03 18:45:03.134812 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.134718    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:03.146817 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.146771    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"]
Nov 03 18:45:03.187478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.187449    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.208536 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.208509    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"]
Nov 03 18:45:03.232930 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.232860    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhmvr\" (UniqueName: \"kubernetes.io/projected/c4b2235a-1853-4d53-8629-cf3bc775fb65-kube-api-access-nhmvr\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.232930 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.232897    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4b2235a-1853-4d53-8629-cf3bc775fb65-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.269808 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.269766    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 18:45:03.269808 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.269775    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lvkbp\""
Nov 03 18:45:03.286492 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.286146    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:03.286841 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.286821    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:03.286935 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.286847    4944 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:45:03.333991 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.333965    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhmvr\" (UniqueName: \"kubernetes.io/projected/c4b2235a-1853-4d53-8629-cf3bc775fb65-kube-api-access-nhmvr\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.334075 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.334016    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4b2235a-1853-4d53-8629-cf3bc775fb65-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.334374 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.334349    4944 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 18:45:03.336237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.336221    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c4b2235a-1853-4d53-8629-cf3bc775fb65-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.377293 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.377192    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:03.391933 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.391907    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhmvr\" (UniqueName: \"kubernetes.io/projected/c4b2235a-1853-4d53-8629-cf3bc775fb65-kube-api-access-nhmvr\") pod \"monitoring-plugin-77f4f76594-d45fs\" (UID: \"c4b2235a-1853-4d53-8629-cf3bc775fb65\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.511373 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.511346    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:03.619641 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.619614    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:03.900869 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.900784    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"]
Nov 03 18:45:03.949035 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:03.949007    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b2235a_1853_4d53_8629_cf3bc775fb65.slice/crio-fabab4ec1e4bbf8a15a9a059c554001db26bbe1410ffb970217a4719e775b675 WatchSource:0}: Error finding container fabab4ec1e4bbf8a15a9a059c554001db26bbe1410ffb970217a4719e775b675: Status 404 returned error can't find the container with id fabab4ec1e4bbf8a15a9a059c554001db26bbe1410ffb970217a4719e775b675
Nov 03 18:45:03.978223 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.978186    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs" event={"ID":"c4b2235a-1853-4d53-8629-cf3bc775fb65","Type":"ContainerStarted","Data":"fabab4ec1e4bbf8a15a9a059c554001db26bbe1410ffb970217a4719e775b675"}
Nov 03 18:45:03.979378 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.979353    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"17ce785e7c4d51b6d6ae3b0e2b5834ea7f1d0da8dff8cd9ac0d3982ac712de56"}
Nov 03 18:45:03.981529 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.981505    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd5jv" event={"ID":"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c","Type":"ContainerStarted","Data":"051d8cbd0fb97abb584e0dd5c7e64ce425d744ba1075a837a0762540a90978f5"}
Nov 03 18:45:03.981634 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.981550    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gd5jv" event={"ID":"a592af2a-0e6b-4455-b71f-bf6c8fe5a14c","Type":"ContainerStarted","Data":"b2c640cd9b6dd65ff67c743f949f40cda78ff03ea4f6b12bdbe18722a6915432"}
Nov 03 18:45:03.982741 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:03.982718    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"0ddfbb9d22ccc0c4836927dd7484f7e97cd311227681c35c98adb2404bd65a22"}
Nov 03 18:45:04.627253 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:04.627178    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:04.989892 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:04.989854    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5q2l5" event={"ID":"ce552fcf-36e3-435c-887d-6ceff825d956","Type":"ContainerStarted","Data":"3ff77ced53359904e9d4ca0b9c953d2a70b5e56b79d5b3cee23fc81e33dfacf5"}
Nov 03 18:45:05.240942 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:05.240848    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:45:05.618310 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:05.618287    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:05.868155 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:05.868069    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:05.994807 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:05.994755    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa" exitCode=0
Nov 03 18:45:05.995291 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:05.994827    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa"}
Nov 03 18:45:06.127346 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:06.127271    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:06.357771 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:06.357743    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ddzdz"
Nov 03 18:45:07.000734 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.000689    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs" event={"ID":"c4b2235a-1853-4d53-8629-cf3bc775fb65","Type":"ContainerStarted","Data":"fa86556e3b663bef1ca5a1ee061798b2125380425a89dae3df1a3c0f34ce0c75"}
Nov 03 18:45:07.112855 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.112780    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs" podStartSLOduration=1.670473589 podStartE2EDuration="4.112768499s" totalImagesPullingTime="2.44229491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:07.111755153 +0000 UTC m=+5.389108735" watchObservedRunningTime="2026-11-03 18:45:07.112768499 +0000 UTC m=+5.390122081"
Nov 03 18:45:07.253230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.253151    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:07.259865 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.259839    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-d45fs"
Nov 03 18:45:07.451262 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.451233    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:07.457389 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.457365    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.506347 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.506279    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 18:45:07.506945 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.506928    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 18:45:07.507508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.507483    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 18:45:07.507596 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.507539    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-du15mn4a5ve4o\""
Nov 03 18:45:07.507702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.507656    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 18:45:07.507702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.507679    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 18:45:07.507780 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.507749    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 18:45:07.517264 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.517231    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 18:45:07.517264 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.517266    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 18:45:07.553168 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.553146    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 18:45:07.553858 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.553834    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 18:45:07.553976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.553859    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n2mh5\""
Nov 03 18:45:07.553976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.553920    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 18:45:07.554109 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.553972    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 18:45:07.554221 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.554201    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 18:45:07.563784 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.563755    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:07.571556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571530    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.571556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571572    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.571556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571679    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.571556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571712    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.571556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571737    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571886    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571935    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.571987    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572014    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572042    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572082    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572167    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572207    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572231    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572257    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572276    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgz4q\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572294    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572313    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572651 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572334    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.572651 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.572351    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.630036 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.630005    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mq4br"
Nov 03 18:45:07.646577 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.646553    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 18:45:07.647396 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.647376    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 18:45:07.648943 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.648926    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:07.672855 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.672829    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.672980 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.672870    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673084 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673022    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673146 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673101    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673146 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673133    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673167    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673190    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673217    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673243    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673275    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673300    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673337    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673372    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673622 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673395    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.673833 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673812    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673928    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673957    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgz4q\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673994    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674413    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674677    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.674716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.673996    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.675111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674738    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.675111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674775    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.675111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674823    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.675111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.674829    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.676511 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.676479    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.676949 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.676925    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678287 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678260    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678392 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678324    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678426    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678745 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678719    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678884 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678865    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.678981 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.678957    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.679414 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.679247    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.679497 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.679473    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.679644 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.679593    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.680898 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.680878    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.680990 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.680969    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.732384 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.732345    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgz4q\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\") pod \"prometheus-k8s-0\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:07.776096 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:07.776022    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:08.006601 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:08.006558    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"abc1999deb79910740612d83f90f642612d4f4f0d2b11989d178ec7dff795056"}
Nov 03 18:45:08.007055 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:08.006640    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"a2db4f4b29d7beb36a7c59185ea52cc9b3e5ec5eca1ba1524d70334be0416091"}
Nov 03 18:45:08.007055 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:08.006683    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"b8587c12307a2597b9c7d693a451287fc6b24e7c7342c49ea7e4f0f52fdf8978"}
Nov 03 18:45:08.219888 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:08.219815    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:08.238768 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:08.238738    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375f880c_bf40_4fae_ac0e_737cd2ffd609.slice/crio-781a9127894d391cc7c2c103bd7bf86ac99965fb8f5da12d130cfcebda44ca55 WatchSource:0}: Error finding container 781a9127894d391cc7c2c103bd7bf86ac99965fb8f5da12d130cfcebda44ca55: Status 404 returned error can't find the container with id 781a9127894d391cc7c2c103bd7bf86ac99965fb8f5da12d130cfcebda44ca55
Nov 03 18:45:08.780014 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:08.779967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:09.013810 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.013751    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"}
Nov 03 18:45:09.014230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.013857    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"}
Nov 03 18:45:09.014230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.013898    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"}
Nov 03 18:45:09.014230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.013937    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"}
Nov 03 18:45:09.014230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.013980    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"}
Nov 03 18:45:09.014230 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.014044    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerStarted","Data":"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"}
Nov 03 18:45:09.017033 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.017004    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"ac7268003987b116b6b8d9616711e09af9bdb8020aaf6236efd3b5e9ef0e4245"}
Nov 03 18:45:09.017149 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.017074    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"6eebd891ff7221626f790492008dbe578ab6e03bc038e41ba6064e4aadbf5bc0"}
Nov 03 18:45:09.017149 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.017113    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2" event={"ID":"618c170c-3333-4548-864b-2e1d78308333","Type":"ContainerStarted","Data":"52826881c600243f3baa458513e5b682600a06b3dec312b14ded1ac9ef3884e3"}
Nov 03 18:45:09.017240 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.017150    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:09.018627 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.018604    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4" exitCode=0
Nov 03 18:45:09.018726 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.018683    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"}
Nov 03 18:45:09.018779 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.018735    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"781a9127894d391cc7c2c103bd7bf86ac99965fb8f5da12d130cfcebda44ca55"}
Nov 03 18:45:09.172167 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.172134    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:09.426716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:09.426630    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:12.094057 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:12.094012    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:45:12.217604 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:12.216636    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:12.217604 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:12.216886    4944 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 18:45:13.039781 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039749    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"}
Nov 03 18:45:13.039925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039835    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"}
Nov 03 18:45:13.039925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039865    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"}
Nov 03 18:45:13.039925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039892    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"}
Nov 03 18:45:13.039925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039916    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"}
Nov 03 18:45:13.040051 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.039938    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerStarted","Data":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"}
Nov 03 18:45:13.165309 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:13.165219    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5012689200000002 podStartE2EDuration="6.165205175s" totalImagesPullingTime="3.637035445s" totalInitContainerRuntime="26.90081ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:45:13.163765802 +0000 UTC m=+11.441119385" watchObservedRunningTime="2026-11-03 18:45:13.165205175 +0000 UTC m=+11.442558757"
Nov 03 18:45:14.378433 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:14.378393    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:14.397203 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:14.397179    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:15.029113 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.029080    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74994bbc58-d8mv2"
Nov 03 18:45:15.047156 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.047129    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:15.062087 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.062066    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:15.287131 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.287042    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:15.558388 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.558311    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:15.900961 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:15.900880    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:16.143354 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:16.143320    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:17.383167 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:17.383107    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:17.636541 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:17.636454    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:19.384659 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:19.384619    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:19.635549 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:19.635456    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:20.360441 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:20.360411    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5g7pb"
Nov 03 18:45:22.311975 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:22.311897    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:22.558393 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:22.558357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:32.096528 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:32.096492    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7dp7q"
Nov 03 18:45:35.223210 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:35.223177    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:45:35.857020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:35.856989    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:36.103267 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:36.103214    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:44.299296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:44.299182    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:44.560824 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:44.560703    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:45.658006 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:45.657976    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:45.906028 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:45.905996    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:47.753701 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:47.753646    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:48.001596 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:48.001550    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:49.862983 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:49.862948    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:50.114422 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:50.114338    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:50.357434 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:50.357368    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:50.861840 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:50.861779    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:51.114440 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.114341    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:45:51.712042 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712010    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 18:45:51.712585 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712557    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="alertmanager" containerID="cri-o://2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123" gracePeriod=120
Nov 03 18:45:51.712650 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712582    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-metric" containerID="cri-o://c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463" gracePeriod=120
Nov 03 18:45:51.712650 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712620    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="prom-label-proxy" containerID="cri-o://fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09" gracePeriod=120
Nov 03 18:45:51.712746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712637    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-web" containerID="cri-o://559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348" gracePeriod=120
Nov 03 18:45:51.712746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712666    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="config-reloader" containerID="cri-o://1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47" gracePeriod=120
Nov 03 18:45:51.712882 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:51.712671    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy" containerID="cri-o://8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695" gracePeriod=120
Nov 03 18:45:52.177953 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177919    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09" exitCode=0
Nov 03 18:45:52.177953 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177944    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463" exitCode=0
Nov 03 18:45:52.177953 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177953    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695" exitCode=0
Nov 03 18:45:52.177953 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177961    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47" exitCode=0
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177970    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123" exitCode=0
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.177984    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"}
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.178045    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"}
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.178071    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"}
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.178096    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"}
Nov 03 18:45:52.178418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.178119    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"}
Nov 03 18:45:52.962381 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:52.962349    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:53.035943 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.035862    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.035943 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.035905    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.035943 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.035931    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036197 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.035965    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036197 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.035988    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036197 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036007    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036197 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036027    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036220    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036281    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036318    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036350    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036377    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036417    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") pod \"b06794b8-5669-4fda-901e-40b6209d79b4\" (UID: \"b06794b8-5669-4fda-901e-40b6209d79b4\") "
Nov 03 18:45:53.036443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036425    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:45:53.036813 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036685    4944 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-main-db\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.036813 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.036781    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:53.037325 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.037284    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:53.039704 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.039673    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "kube-api-access-pnzrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:53.039821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.039674    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.039821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.039805    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.039940 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.039908    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.040066 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.040036    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.040169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.040096    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:45:53.040169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.040119    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.040807 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.040773    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:53.044647 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.044623    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.050516 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.050490    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config" pod "b06794b8-5669-4fda-901e-40b6209d79b4" (UID: "b06794b8-5669-4fda-901e-40b6209d79b4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:53.137687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137650    4944 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137681    4944 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-metrics-client-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137695    4944 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b06794b8-5669-4fda-901e-40b6209d79b4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137711    4944 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b06794b8-5669-4fda-901e-40b6209d79b4-config-out\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137722    4944 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-tls-assets\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137735    4944 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-cluster-tls-config\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137748    4944 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137762    4944 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-main-tls\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137774    4944 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-web-config\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137809    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnzrg\" (UniqueName: \"kubernetes.io/projected/b06794b8-5669-4fda-901e-40b6209d79b4-kube-api-access-pnzrg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137823    4944 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.137932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.137836    4944 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b06794b8-5669-4fda-901e-40b6209d79b4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:53.185059 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.185029    4944 generic.go:356] "Generic (PLEG): container finished" podID="b06794b8-5669-4fda-901e-40b6209d79b4" containerID="559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348" exitCode=0
Nov 03 18:45:53.185441 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.185093    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"}
Nov 03 18:45:53.185441 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.185122    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 18:45:53.185441 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.185148    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b06794b8-5669-4fda-901e-40b6209d79b4","Type":"ContainerDied","Data":"0ddfbb9d22ccc0c4836927dd7484f7e97cd311227681c35c98adb2404bd65a22"}
Nov 03 18:45:53.185441 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.185191    4944 scope.go:122] "RemoveContainer" containerID="fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"
Nov 03 18:45:53.193667 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.193652    4944 scope.go:122] "RemoveContainer" containerID="c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"
Nov 03 18:45:53.200098 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.200083    4944 scope.go:122] "RemoveContainer" containerID="8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"
Nov 03 18:45:53.206455 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.206439    4944 scope.go:122] "RemoveContainer" containerID="559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"
Nov 03 18:45:53.212700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.212682    4944 scope.go:122] "RemoveContainer" containerID="1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"
Nov 03 18:45:53.219517 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.219500    4944 scope.go:122] "RemoveContainer" containerID="2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"
Nov 03 18:45:53.226806 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.226772    4944 scope.go:122] "RemoveContainer" containerID="71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa"
Nov 03 18:45:53.236909 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.236894    4944 scope.go:122] "RemoveContainer" containerID="fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"
Nov 03 18:45:53.237225 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.237197    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09\": container with ID starting with fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09 not found: ID does not exist" containerID="fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"
Nov 03 18:45:53.237308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237233    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09"} err="failed to get container status \"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09\": rpc error: code = NotFound desc = could not find container \"fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09\": container with ID starting with fbe1d9b6fbcea89970fced874402b7c87c1e5e6fea7487ee8824c6c8d4953b09 not found: ID does not exist"
Nov 03 18:45:53.237308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237268    4944 scope.go:122] "RemoveContainer" containerID="c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"
Nov 03 18:45:53.237554 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.237530    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463\": container with ID starting with c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463 not found: ID does not exist" containerID="c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"
Nov 03 18:45:53.237600 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237566    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463"} err="failed to get container status \"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463\": rpc error: code = NotFound desc = could not find container \"c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463\": container with ID starting with c36fb630eccc56a95f0c225c12422f5642973bd5c27c764a013757315bf70463 not found: ID does not exist"
Nov 03 18:45:53.237600 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237591    4944 scope.go:122] "RemoveContainer" containerID="8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"
Nov 03 18:45:53.237881 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.237860    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695\": container with ID starting with 8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695 not found: ID does not exist" containerID="8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"
Nov 03 18:45:53.237936 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237890    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695"} err="failed to get container status \"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695\": rpc error: code = NotFound desc = could not find container \"8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695\": container with ID starting with 8a557832d5fe0a8a2f49974ca72cc55c5e38b5a446e132859374767dc43c0695 not found: ID does not exist"
Nov 03 18:45:53.237936 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.237909    4944 scope.go:122] "RemoveContainer" containerID="559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"
Nov 03 18:45:53.238194 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.238174    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348\": container with ID starting with 559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348 not found: ID does not exist" containerID="559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"
Nov 03 18:45:53.238261 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238199    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348"} err="failed to get container status \"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348\": rpc error: code = NotFound desc = could not find container \"559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348\": container with ID starting with 559b45d97060497eb95ac730cfc4da0cf5fa2775be3326a52521f19521f26348 not found: ID does not exist"
Nov 03 18:45:53.238261 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238213    4944 scope.go:122] "RemoveContainer" containerID="1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"
Nov 03 18:45:53.238475 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.238455    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47\": container with ID starting with 1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47 not found: ID does not exist" containerID="1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"
Nov 03 18:45:53.238546 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238478    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47"} err="failed to get container status \"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47\": rpc error: code = NotFound desc = could not find container \"1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47\": container with ID starting with 1101b6f84306ec87c475da3fe69b2d55dd733913afa687bce84dc374293c8b47 not found: ID does not exist"
Nov 03 18:45:53.238546 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238491    4944 scope.go:122] "RemoveContainer" containerID="2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"
Nov 03 18:45:53.238754 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.238737    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123\": container with ID starting with 2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123 not found: ID does not exist" containerID="2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"
Nov 03 18:45:53.238809 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238758    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123"} err="failed to get container status \"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123\": rpc error: code = NotFound desc = could not find container \"2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123\": container with ID starting with 2149822401b49045c09911356daa58a6ab95dabf4e3ee5d38851749422e3d123 not found: ID does not exist"
Nov 03 18:45:53.238809 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.238770    4944 scope.go:122] "RemoveContainer" containerID="71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa"
Nov 03 18:45:53.239093 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:53.239079    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa\": container with ID starting with 71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa not found: ID does not exist" containerID="71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa"
Nov 03 18:45:53.239156 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.239096    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa"} err="failed to get container status \"71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa\": rpc error: code = NotFound desc = could not find container \"71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa\": container with ID starting with 71b7adb2c5a43e58674272b7849dcb84e8d63129150ff5d9fbbddfa6d682b2aa not found: ID does not exist"
Nov 03 18:45:53.341969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.341890    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 18:45:53.345413 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.345382    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 18:45:53.932742 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:53.932701    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" path="/var/lib/kubelet/pods/b06794b8-5669-4fda-901e-40b6209d79b4/volumes"
Nov 03 18:45:57.453954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.453923    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:57.454365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454218    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="prometheus" containerID="cri-o://bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791" gracePeriod=600
Nov 03 18:45:57.454365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454250    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy" containerID="cri-o://fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29" gracePeriod=600
Nov 03 18:45:57.454365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454301    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="thanos-sidecar" containerID="cri-o://18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b" gracePeriod=600
Nov 03 18:45:57.454365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454319    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-web" containerID="cri-o://ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7" gracePeriod=600
Nov 03 18:45:57.454365 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454326    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="config-reloader" containerID="cri-o://22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302" gracePeriod=600
Nov 03 18:45:57.454645 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.454302    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93" gracePeriod=600
Nov 03 18:45:57.700746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.700719    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:57.771802 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.771761    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.771966 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.771830    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.771966 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.771859    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.771966 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.771892    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772021    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772063    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772098    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.772290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772111    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772213    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772245    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772260    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.772290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772272    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772289    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772400    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772434    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772460    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772491    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772529    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772871 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772576    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772871 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772621    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772871 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772654    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772871 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772687    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.772871 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772718    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"375f880c-bf40-4fae-ac0e-737cd2ffd609\" (UID: \"375f880c-bf40-4fae-ac0e-737cd2ffd609\") "
Nov 03 18:45:57.773135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772949    4944 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.773135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772969    4944 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.773135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.772986    4944 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.774446 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774421    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "kube-api-access-fgz4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774628    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774702    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774767    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774831    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.774954    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.775098    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.775135    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.775317 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.775224    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 18:45:57.775918 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.775889    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.776035 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.776014    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.776306 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.776268    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:45:57.776773 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.776749    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.777023 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.776993    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.777023 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.777008    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.777234 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.777213    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 18:45:57.786484 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.786462    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config" pod "375f880c-bf40-4fae-ac0e-737cd2ffd609" (UID: "375f880c-bf40-4fae-ac0e-737cd2ffd609"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:45:57.873420 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873389    4944 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-db\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873420 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873414    4944 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873420 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873426    4944 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-metrics-client-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873435    4944 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873446    4944 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-config\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873455    4944 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-kube-rbac-proxy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873463    4944 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-tls-assets\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873472    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgz4q\" (UniqueName: \"kubernetes.io/projected/375f880c-bf40-4fae-ac0e-737cd2ffd609-kube-api-access-fgz4q\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873480    4944 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873489    4944 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873498    4944 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-grpc-tls\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873509    4944 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-web-config\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873518    4944 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873527    4944 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/375f880c-bf40-4fae-ac0e-737cd2ffd609-configmap-metrics-client-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873535    4944 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/375f880c-bf40-4fae-ac0e-737cd2ffd609-config-out\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873544    4944 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:57.873639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:57.873553    4944 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/375f880c-bf40-4fae-ac0e-737cd2ffd609-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205051    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205076    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205084    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205090    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205095    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205100    4944 generic.go:356] "Generic (PLEG): container finished" podID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791" exitCode=0
Nov 03 18:45:58.205138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205131    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205130    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205266    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205304    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205329    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205331    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205355    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205387    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"}
Nov 03 18:45:58.205578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.205410    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"375f880c-bf40-4fae-ac0e-737cd2ffd609","Type":"ContainerDied","Data":"781a9127894d391cc7c2c103bd7bf86ac99965fb8f5da12d130cfcebda44ca55"}
Nov 03 18:45:58.212626 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.212608    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.219143 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.219120    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.225489 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.225472    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.235822 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.235776    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.242312 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.242288    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.249068 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.249050    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.255478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.255463    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.255732 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.255713    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.255817 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.255740    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.255817 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.255756    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.256096 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.256079    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.256140 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256101    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.256140 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256114    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.256353 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.256331    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.256393 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256360    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.256393 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256375    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.256587 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.256569    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.256683 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256589    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.256683 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256607    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.256880 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.256859    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.256944 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256886    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.256944 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.256900    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.257144 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.257127    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.257182 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257150    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.257182 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257170    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.257411 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:45:58.257393    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.257472 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257414    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.257472 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257427    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.257660 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257641    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.257721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257662    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.257924 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257906    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.257975 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.257925    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.258206 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258190    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.258248 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258206    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.258388 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258373    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.258433 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258388    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.258567 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258551    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.258606 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258566    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.258730 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258715    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.258770 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258730    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.258973 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258955    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.258973 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.258973    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.259183 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259170    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.259183 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259182    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.259397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259371    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.259438 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259397    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.259593 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259579    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.259637 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259592    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.259825 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259807    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.259875 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259826    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.260007 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.259994    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.260043 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260007    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.260226 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260206    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.260276 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260226    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.260429 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260413    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.260429 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260426    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.260631 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260609    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.260631 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260630    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.260857 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260835    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.260913 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.260856    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.261067 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261051    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.261106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261068    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.261256 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261242    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.261296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261256    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.261440 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261423    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.261480 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261441    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.261626 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261612    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.261669 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261625    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.261859 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261843    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.261859 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.261859    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.262063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262048    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.262063 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262062    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.262270 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262250    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.262312 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262270    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.262485 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262466    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.262526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262484    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.262679 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262660    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.262715 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262680    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.262904 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262883    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.262950 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.262905    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.263108 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263094    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.263148 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263109    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.263322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263304    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.263361 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263323    4944 scope.go:122] "RemoveContainer" containerID="e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"
Nov 03 18:45:58.263532 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263515    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93"} err="failed to get container status \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": rpc error: code = NotFound desc = could not find container \"e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93\": container with ID starting with e148131ad5a717a2b91d7d1bea578201ef982018bb65cdf58d4f32e3e79e1e93 not found: ID does not exist"
Nov 03 18:45:58.263572 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263531    4944 scope.go:122] "RemoveContainer" containerID="fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"
Nov 03 18:45:58.263758 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263740    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29"} err="failed to get container status \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": rpc error: code = NotFound desc = could not find container \"fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29\": container with ID starting with fa82a224aa3612757457961d255b3cf38ce1233360844f453692818a1667bf29 not found: ID does not exist"
Nov 03 18:45:58.263822 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263759    4944 scope.go:122] "RemoveContainer" containerID="ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"
Nov 03 18:45:58.264000 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.263982    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7"} err="failed to get container status \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": rpc error: code = NotFound desc = could not find container \"ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7\": container with ID starting with ccae2256b123091cd0ea20b0b57848930e9c7406e04422277c277f27a53ce4a7 not found: ID does not exist"
Nov 03 18:45:58.264038 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264002    4944 scope.go:122] "RemoveContainer" containerID="18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"
Nov 03 18:45:58.264204 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264185    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b"} err="failed to get container status \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": rpc error: code = NotFound desc = could not find container \"18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b\": container with ID starting with 18f52fd6406b0fdca1c3c3d9235d9b3d87829655c13e88a60f2b7b8b6968e30b not found: ID does not exist"
Nov 03 18:45:58.264252 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264205    4944 scope.go:122] "RemoveContainer" containerID="22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"
Nov 03 18:45:58.264447 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264426    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302"} err="failed to get container status \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": rpc error: code = NotFound desc = could not find container \"22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302\": container with ID starting with 22048a9dac2c7825c75a7be000ae0ded5e878ac004eb5b715e947c44e0aae302 not found: ID does not exist"
Nov 03 18:45:58.264447 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264444    4944 scope.go:122] "RemoveContainer" containerID="bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"
Nov 03 18:45:58.264687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264671    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791"} err="failed to get container status \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": rpc error: code = NotFound desc = could not find container \"bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791\": container with ID starting with bca8880e2938c293b014c1b9a8b641ae2b870b5854bbea194c8e4a756888e791 not found: ID does not exist"
Nov 03 18:45:58.264687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264687    4944 scope.go:122] "RemoveContainer" containerID="b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"
Nov 03 18:45:58.264934 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.264912    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4"} err="failed to get container status \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": rpc error: code = NotFound desc = could not find container \"b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4\": container with ID starting with b00cc98d613ef0727561c89426fdf7bb3a43931bf082d14538f7017ba2dcffa4 not found: ID does not exist"
Nov 03 18:45:58.369448 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.369419    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:58.416424 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.416391    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:58.416960 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.416944    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:58.417201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417190    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="alertmanager"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417203    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="alertmanager"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417212    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="config-reloader"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417216    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="config-reloader"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417224    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="prometheus"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417229    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="prometheus"
Nov 03 18:45:58.417237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417237    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-thanos"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417242    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-thanos"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417248    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-metric"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417252    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-metric"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417260    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="thanos-sidecar"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417264    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="thanos-sidecar"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417270    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417275    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417284    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417288    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417294    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417298    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417305    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417309    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417319    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417323    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417329    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="prom-label-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417333    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="prom-label-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417339    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="init-config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417343    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="init-config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417351    4944 cpu_manager.go:414] "containerMap: removing container" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="init-config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417355    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="init-config-reloader"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417392    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417399    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy"
Nov 03 18:45:58.417397 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417405    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="config-reloader"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417411    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417417    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="prom-label-proxy"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417425    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="config-reloader"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417431    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="alertmanager"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417436    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="prometheus"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417441    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-thanos"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417445    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b06794b8-5669-4fda-901e-40b6209d79b4" containerName="kube-rbac-proxy-metric"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417450    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="thanos-sidecar"
Nov 03 18:45:58.418020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.417454    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" containerName="kube-rbac-proxy-web"
Nov 03 18:45:58.421366 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.421351    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.471002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.470973    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 03 18:45:58.471477 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.471363    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 18:45:58.471477 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.471462    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 03 18:45:58.471704 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.471686    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 03 18:45:58.477940 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.477917    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478069 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.477971    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478069 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478005    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478175 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478078    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478175 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478106    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478175 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478122    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478175    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478201    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478220    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478236    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478274    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478306    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478416    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478439    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478474    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478510    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478527    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4phq\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-kube-api-access-b4phq\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478542    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478566    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.478612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.478593    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.517968 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.517945    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 03 18:45:58.518202 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518152    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 03 18:45:58.518202 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518199    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 03 18:45:58.518358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518199    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 03 18:45:58.518358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518199    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 03 18:45:58.518358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518302    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 03 18:45:58.518358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518318    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 03 18:45:58.518522 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518417    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 03 18:45:58.518555 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518539    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-du15mn4a5ve4o\""
Nov 03 18:45:58.518673 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518659    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-n2mh5\""
Nov 03 18:45:58.518849 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518832    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 03 18:45:58.518945 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.518917    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 03 18:45:58.519064 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.519051    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 03 18:45:58.532574 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.532553    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:58.579251 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579221    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579371 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579256    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579371 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579283    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579371 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579309    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579371 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579333    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579371 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579367    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579394    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579418    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579439    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579460    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579484    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579513    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579538    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579568    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.579617 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579599    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579622    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4phq\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-kube-api-access-b4phq\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579643    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579668    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579693    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.579726    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.580111    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.580290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.580112    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.581964 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.581318    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.581964 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.581575    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.581964 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.581728    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.582540 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.582510    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.582665 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.582636    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.583011 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.582959    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.583011 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.582995    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.583168 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.583035    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.583878 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.583856    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.584315 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.584245    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585015 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.584563    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585015 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.584974    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585155 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.585073    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-config\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585155 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.585105    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.585458    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.585709 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.585690    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.586264 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.586242    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.637528 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.637500    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4phq\" (UniqueName: \"kubernetes.io/projected/f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a-kube-api-access-b4phq\") pod \"prometheus-k8s-0\" (UID: \"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:58.739243 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:58.739165    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:45:59.171054 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:59.171020    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 03 18:45:59.179235 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:45:59.179206    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82db8a5_a394_4bf9_b9a8_27cd5e6fd80a.slice/crio-e14465352d827aa5ff386cf0087f26f2edf27935e37d957cc2aca9439b8dc176 WatchSource:0}: Error finding container e14465352d827aa5ff386cf0087f26f2edf27935e37d957cc2aca9439b8dc176: Status 404 returned error can't find the container with id e14465352d827aa5ff386cf0087f26f2edf27935e37d957cc2aca9439b8dc176
Nov 03 18:45:59.210352 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:59.210324    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"e14465352d827aa5ff386cf0087f26f2edf27935e37d957cc2aca9439b8dc176"}
Nov 03 18:45:59.933607 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:45:59.933571    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375f880c-bf40-4fae-ac0e-737cd2ffd609" path="/var/lib/kubelet/pods/375f880c-bf40-4fae-ac0e-737cd2ffd609/volumes"
Nov 03 18:46:00.214821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:00.214724    4944 generic.go:356] "Generic (PLEG): container finished" podID="f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a" containerID="618fd481ce4e5765a2c7d1bd984c4654704cfd945689b0de08e8b28ac7a1cbf4" exitCode=0
Nov 03 18:46:00.214821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:00.214768    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerDied","Data":"618fd481ce4e5765a2c7d1bd984c4654704cfd945689b0de08e8b28ac7a1cbf4"}
Nov 03 18:46:01.221588 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221548    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"2eb29f2e902415ecc67249b8ac2868c2f5e8288850843d68fd794c8adcc0ebf3"}
Nov 03 18:46:01.222002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221614    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"806c150ecafe934a10cefaa1d55a62825f72f6358d1d2b4dc5e794a440b2564a"}
Nov 03 18:46:01.222002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221640    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"40dff8b841e9bd273a39a3a3a09f01190694b2ff716d0b0f10213ecd3161663d"}
Nov 03 18:46:01.222002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221679    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"6ebe1a174fb318ec7c1e5a647b37a4665ba690346fea0dc9be4e31d2660cce7a"}
Nov 03 18:46:01.222002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221705    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"56da51179dd75b6885d3e516362a46a56a1949b0d8e4d2360f5a57fade06554d"}
Nov 03 18:46:01.222002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.221729    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a","Type":"ContainerStarted","Data":"f036d3fe873522eb5051940e622a9eb3a3045260a6773c087bf5929d414f15d6"}
Nov 03 18:46:01.348513 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:01.348460    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.332735275 podStartE2EDuration="3.348446287s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.711012ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:46:01.347445282 +0000 UTC m=+59.624798866" watchObservedRunningTime="2026-11-03 18:46:01.348446287 +0000 UTC m=+59.625799870"
Nov 03 18:46:03.739846 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:03.739783    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:46:05.223263 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:46:05.223222    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:46:15.759078 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:15.759046    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:46:35.223249 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:46:35.223208    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:46:56.083639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:56.083595    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:46:56.098918 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:56.098893    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:46:56.418632 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:46:56.418564    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 03 18:47:01.014409 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:01.014379    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:01.262442 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:01.262404    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:05.223135 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:47:05.223101    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:47:16.112710 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:16.112635    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:18.215125 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:18.215088    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:20.514664 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:20.514626    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:20.795669 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:20.795592    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:31.890327 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:31.890291    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:35.222187 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:47:35.222146    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:47:47.579726 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:47.579695    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:48.929898 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:48.929865    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:50.797771 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:50.797739    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:51.054463 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:51.054390    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:55.006880 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:55.006842    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:55.282089 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:55.282006    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:55.540299 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:55.540222    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:55.984579 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:55.984508    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:56.237917 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:56.237843    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:57.278769 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:57.278731    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:47:57.962306 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:47:57.962270    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:04.606175 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:04.606141    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:04.854960 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:04.854926    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:05.222770 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:48:05.222736    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:48:05.863207 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:05.863166    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:07.274425 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.274381    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-t8t6l"]
Nov 03 18:48:07.276576 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.276559    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.325386 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.325355    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 18:48:07.361449 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.361410    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-bound-sa-token\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.361618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.361537    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4klw\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-kube-api-access-r4klw\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.371487 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.371458    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 18:48:07.371594 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.371506    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bc56g\""
Nov 03 18:48:07.372328 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.372309    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t8t6l"]
Nov 03 18:48:07.461972 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.461935    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-bound-sa-token\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.462143 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.462005    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4klw\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-kube-api-access-r4klw\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.519019 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.518988    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-bound-sa-token\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.519279 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.519257    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4klw\" (UniqueName: \"kubernetes.io/projected/874927aa-ddda-4114-b80d-a198ad916532-kube-api-access-r4klw\") pod \"cert-manager-597b95cb84-t8t6l\" (UID: \"874927aa-ddda-4114-b80d-a198ad916532\") " pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.596280 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.596203    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-t8t6l"
Nov 03 18:48:07.961234 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:07.961155    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-t8t6l"]
Nov 03 18:48:07.969979 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:48:07.969946    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod874927aa_ddda_4114_b80d_a198ad916532.slice/crio-dc57b1f9edb670bf0bc34b482830114ab38ec3450ace7cf8dd723e51f307c211 WatchSource:0}: Error finding container dc57b1f9edb670bf0bc34b482830114ab38ec3450ace7cf8dd723e51f307c211: Status 404 returned error can't find the container with id dc57b1f9edb670bf0bc34b482830114ab38ec3450ace7cf8dd723e51f307c211
Nov 03 18:48:08.635249 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:08.635208    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t8t6l" event={"ID":"874927aa-ddda-4114-b80d-a198ad916532","Type":"ContainerStarted","Data":"dc57b1f9edb670bf0bc34b482830114ab38ec3450ace7cf8dd723e51f307c211"}
Nov 03 18:48:08.891830 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:08.891736    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:09.148645 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:09.148563    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:13.654975 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:13.654941    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-t8t6l" event={"ID":"874927aa-ddda-4114-b80d-a198ad916532","Type":"ContainerStarted","Data":"aa73cece0145c466a7b9510e68372e002107b00257280bfa289bf3058072c770"}
Nov 03 18:48:13.763907 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:13.763858    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-t8t6l" podStartSLOduration=1.870187358 podStartE2EDuration="6.763843067s" totalImagesPullingTime="4.893655709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:13.76321755 +0000 UTC m=+192.040571134" watchObservedRunningTime="2026-11-03 18:48:13.763843067 +0000 UTC m=+192.041196649"
Nov 03 18:48:16.512234 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:16.512199    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:19.768257 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:19.768213    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:33.827674 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:33.827640    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:34.080482 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.080401    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:34.604679 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.604650    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:34.752070 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.752039    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"]
Nov 03 18:48:34.754666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.754650    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.868570 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868478    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tqcfl\""
Nov 03 18:48:34.868570 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868503    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 18:48:34.868570 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868523    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 18:48:34.869157 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868575    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 18:48:34.869157 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868582    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 18:48:34.869157 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.868649    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Nov 03 18:48:34.869157 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869117    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.869157 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869150    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.869387 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869182    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qr6\" (UniqueName: \"kubernetes.io/projected/0f9f5095-7613-41d5-b161-a58fac039e4a-kube-api-access-s5qr6\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.869387 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869312    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.869387 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869346    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.869387 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.869370    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.875972 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.875944    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"]
Nov 03 18:48:34.970431 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970398    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.970638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970470    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qr6\" (UniqueName: \"kubernetes.io/projected/0f9f5095-7613-41d5-b161-a58fac039e4a-kube-api-access-s5qr6\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.970638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970550    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.970638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970595    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.970638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970618    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.970897 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.970693    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.971164 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.971138    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-manifests\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.973070 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.973041    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.973168 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.973074    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0f9f5095-7613-41d5-b161-a58fac039e4a-tmp\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.973168 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.973051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:34.973417 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:34.973402    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0f9f5095-7613-41d5-b161-a58fac039e4a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:35.029937 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.029905    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:35.032395 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.032367    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qr6\" (UniqueName: \"kubernetes.io/projected/0f9f5095-7613-41d5-b161-a58fac039e4a-kube-api-access-s5qr6\") pod \"opendatahub-operator-controller-manager-5ff9d8584c-fcklg\" (UID: \"0f9f5095-7613-41d5-b161-a58fac039e4a\") " pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:35.067384 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.067348    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:35.222866 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:48:35.222761    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:48:35.285429 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.285401    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:35.540606 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.540572    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:35.557072 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.557044    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"]
Nov 03 18:48:35.565408 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:48:35.565376    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9f5095_7613_41d5_b161_a58fac039e4a.slice/crio-53ae0ce785433ce4f2e591e6b52617e06792f0d2a1d2b335d95a70e231df2dbc WatchSource:0}: Error finding container 53ae0ce785433ce4f2e591e6b52617e06792f0d2a1d2b335d95a70e231df2dbc: Status 404 returned error can't find the container with id 53ae0ce785433ce4f2e591e6b52617e06792f0d2a1d2b335d95a70e231df2dbc
Nov 03 18:48:35.725824 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:35.725774    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg" event={"ID":"0f9f5095-7613-41d5-b161-a58fac039e4a","Type":"ContainerStarted","Data":"53ae0ce785433ce4f2e591e6b52617e06792f0d2a1d2b335d95a70e231df2dbc"}
Nov 03 18:48:38.732322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:38.732278    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:38.982829 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:38.982720    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:41.622555 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:41.622471    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:44.447864 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:44.447836    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:44.697488 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:44.697404    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:44.759598 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:44.759565    4944 generic.go:356] "Generic (PLEG): container finished" podID="0f9f5095-7613-41d5-b161-a58fac039e4a" containerID="9eb53ffd997f6fc2acdd9faedfaf3e1b13be113c3aca57ffd736d5cbfda7e3ce" exitCode=0
Nov 03 18:48:44.759755 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:44.759606    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg" event={"ID":"0f9f5095-7613-41d5-b161-a58fac039e4a","Type":"ContainerDied","Data":"9eb53ffd997f6fc2acdd9faedfaf3e1b13be113c3aca57ffd736d5cbfda7e3ce"}
Nov 03 18:48:44.990812 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:44.990772    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:45.764890 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:45.764852    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg" event={"ID":"0f9f5095-7613-41d5-b161-a58fac039e4a","Type":"ContainerStarted","Data":"6c611f752eb1e78a8ba2439f050d71af86f961d7a76732b1617eac89dadeef19"}
Nov 03 18:48:45.765359 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:45.764980    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:45.986438 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:45.986306    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg" podStartSLOduration=2.348983161 podStartE2EDuration="11.986285052s" totalImagesPullingTime="9.217636084s" totalInitContainerRuntime="419.665807ms" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:48:45.940068322 +0000 UTC m=+224.217421905" watchObservedRunningTime="2026-11-03 18:48:45.986285052 +0000 UTC m=+224.263638637"
Nov 03 18:48:53.555229 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.555192    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:53.773721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.773685    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"]
Nov 03 18:48:53.780008 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.779985    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:53.829214 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.829129    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Nov 03 18:48:53.829329 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.829299    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Nov 03 18:48:53.875561 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.875527    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Nov 03 18:48:53.875760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.875742    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Nov 03 18:48:53.875862 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.875844    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-v4rxv\""
Nov 03 18:48:53.876055 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.876041    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Nov 03 18:48:53.879176 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.879156    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"]
Nov 03 18:48:53.920560 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.920530    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:53.920714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.920568    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pvw\" (UniqueName: \"kubernetes.io/projected/810509b0-f36d-4b92-8ea8-f0840e8af0ce-kube-api-access-n7pvw\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:53.920714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.920613    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/810509b0-f36d-4b92-8ea8-f0840e8af0ce-manager-config\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:53.920839 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:53.920722    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-metrics-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.021218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.021182    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/810509b0-f36d-4b92-8ea8-f0840e8af0ce-manager-config\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.021218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.021218    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-metrics-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.021458 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.021249    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.021458 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.021272    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pvw\" (UniqueName: \"kubernetes.io/projected/810509b0-f36d-4b92-8ea8-f0840e8af0ce-kube-api-access-n7pvw\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.021936 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.021911    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/810509b0-f36d-4b92-8ea8-f0840e8af0ce-manager-config\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.023761 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.023740    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.023864 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.023749    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/810509b0-f36d-4b92-8ea8-f0840e8af0ce-metrics-cert\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.079634 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.079557    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pvw\" (UniqueName: \"kubernetes.io/projected/810509b0-f36d-4b92-8ea8-f0840e8af0ce-kube-api-access-n7pvw\") pod \"lws-controller-manager-75cc556d68-fxzxh\" (UID: \"810509b0-f36d-4b92-8ea8-f0840e8af0ce\") " pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.091442 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.091414    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:48:54.461842 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.461739    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"]
Nov 03 18:48:54.470196 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:48:54.470166    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810509b0_f36d_4b92_8ea8_f0840e8af0ce.slice/crio-878edc0729ace4f9819a242960fccd3efb78a3fe88420615905f271d3eeecc6c WatchSource:0}: Error finding container 878edc0729ace4f9819a242960fccd3efb78a3fe88420615905f271d3eeecc6c: Status 404 returned error can't find the container with id 878edc0729ace4f9819a242960fccd3efb78a3fe88420615905f271d3eeecc6c
Nov 03 18:48:54.795243 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:54.795202    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh" event={"ID":"810509b0-f36d-4b92-8ea8-f0840e8af0ce","Type":"ContainerStarted","Data":"878edc0729ace4f9819a242960fccd3efb78a3fe88420615905f271d3eeecc6c"}
Nov 03 18:48:56.770467 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:56.770438    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5ff9d8584c-fcklg"
Nov 03 18:48:59.748896 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:59.748860    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:48:59.998643 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:48:59.998602    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:00.530263 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:00.530207    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:00.810421 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:00.810301    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:01.875824 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.875782    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"]
Nov 03 18:49:01.880677 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.880660    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:01.912156 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.912130    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:01.933206 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.933176    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 18:49:01.964038 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.964009    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 18:49:01.977348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.977309    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 18:49:01.978342 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.978323    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"]
Nov 03 18:49:01.978653 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.978630    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c9ztg\""
Nov 03 18:49:01.978773 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.978666    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 18:49:01.987329 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.987305    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdgqh\" (UniqueName: \"kubernetes.io/projected/65da9844-a098-4c47-8c70-6e1b8b923b25-kube-api-access-sdgqh\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:01.987436 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.987359    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da9844-a098-4c47-8c70-6e1b8b923b25-tmp\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:01.987498 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:01.987446    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65da9844-a098-4c47-8c70-6e1b8b923b25-tls-certs\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.088879 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.088847    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdgqh\" (UniqueName: \"kubernetes.io/projected/65da9844-a098-4c47-8c70-6e1b8b923b25-kube-api-access-sdgqh\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.089030 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.088898    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da9844-a098-4c47-8c70-6e1b8b923b25-tmp\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.089030 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.088917    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65da9844-a098-4c47-8c70-6e1b8b923b25-tls-certs\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.091241 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.091209    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65da9844-a098-4c47-8c70-6e1b8b923b25-tmp\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.091473 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.091456    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65da9844-a098-4c47-8c70-6e1b8b923b25-tls-certs\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.146534 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.146455    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdgqh\" (UniqueName: \"kubernetes.io/projected/65da9844-a098-4c47-8c70-6e1b8b923b25-kube-api-access-sdgqh\") pod \"kube-auth-proxy-86dc56b666-vbh86\" (UID: \"65da9844-a098-4c47-8c70-6e1b8b923b25\") " pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.183372 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.183340    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:02.273155 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.273121    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"
Nov 03 18:49:02.639141 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.639103    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-86dc56b666-vbh86"]
Nov 03 18:49:02.647178 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:49:02.647155    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da9844_a098_4c47_8c70_6e1b8b923b25.slice/crio-854180ee8d0e895a2bc6eed45a3a423ce9ea627edd2eb562c2805e0b6473ed8a WatchSource:0}: Error finding container 854180ee8d0e895a2bc6eed45a3a423ce9ea627edd2eb562c2805e0b6473ed8a: Status 404 returned error can't find the container with id 854180ee8d0e895a2bc6eed45a3a423ce9ea627edd2eb562c2805e0b6473ed8a
Nov 03 18:49:02.824874 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.824841    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh" event={"ID":"810509b0-f36d-4b92-8ea8-f0840e8af0ce","Type":"ContainerStarted","Data":"779b6c81da7c994ff02db2e28e1a986d51ee15fab8f697f1ab4d302af8130485"}
Nov 03 18:49:02.825068 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.824895    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:49:02.825998 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:02.825973    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86" event={"ID":"65da9844-a098-4c47-8c70-6e1b8b923b25","Type":"ContainerStarted","Data":"854180ee8d0e895a2bc6eed45a3a423ce9ea627edd2eb562c2805e0b6473ed8a"}
Nov 03 18:49:03.633337 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:03.633300    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:03.884673 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:03.884592    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:04.874234 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:04.874203    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:05.223628 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:49:05.223536    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:49:07.845829 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:07.845785    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86" event={"ID":"65da9844-a098-4c47-8c70-6e1b8b923b25","Type":"ContainerStarted","Data":"27b5517f35a5517f30bcc4a7b005d29f709c43bd0f4696cb497fd8e4d1f7eded"}
Nov 03 18:49:07.988109 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:07.988048    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh" podStartSLOduration=7.594240127 podStartE2EDuration="14.988033708s" totalImagesPullingTime="7.393793581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:02.935178222 +0000 UTC m=+241.212531808" watchObservedRunningTime="2026-11-03 18:49:07.988033708 +0000 UTC m=+246.265387293"
Nov 03 18:49:07.988363 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:07.988337    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-86dc56b666-vbh86" podStartSLOduration=1.8661052470000001 podStartE2EDuration="6.988330681s" totalImagesPullingTime="5.122225434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:49:07.987202626 +0000 UTC m=+246.264556208" watchObservedRunningTime="2026-11-03 18:49:07.988330681 +0000 UTC m=+246.265684263"
Nov 03 18:49:09.493537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:09.493489    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:10.075734 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:10.075699    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:10.672246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:10.672210    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:10.933714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:10.933639    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:11.181147 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:11.181115    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:11.614076 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:11.614039    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:12.419967 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:12.419935    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:12.672660 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:12.672583    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:13.831771 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:13.831740    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75cc556d68-fxzxh"
Nov 03 18:49:16.839651 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:16.839617    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:26.803297 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:26.803259    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:27.322296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:27.322268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:30.555564 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:30.555530    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:30.807625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:49:30.807550    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:49:35.222640 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:49:35.222601    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:50:01.945748 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:01.945719    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:50:01.946264 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:01.946017    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:50:05.223516 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:50:05.223478    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:50:11.826530 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:11.826455    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:12.131827 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:12.131737    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:17.223418 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:17.223377    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:24.231206 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:24.231169    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:24.479002 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:24.478960    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:24.728200 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:24.728169    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:24.981828 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:24.981723    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:25.230806 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.230754    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tx2vn"]
Nov 03 18:50:25.233094 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.233034    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.243357 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.243334    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:25.248414 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.248386    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144204e6-4a23-4dfe-9686-e6da6c450145-tls-certs\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.248478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.248423    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgl9q\" (UniqueName: \"kubernetes.io/projected/144204e6-4a23-4dfe-9686-e6da6c450145-kube-api-access-lgl9q\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.288601 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.288578    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tx2vn"]
Nov 03 18:50:25.330314 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.330284    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-x6np2\""
Nov 03 18:50:25.330425 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.330330    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 18:50:25.349344 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.349315    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144204e6-4a23-4dfe-9686-e6da6c450145-tls-certs\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.349456 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.349344    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgl9q\" (UniqueName: \"kubernetes.io/projected/144204e6-4a23-4dfe-9686-e6da6c450145-kube-api-access-lgl9q\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.351796 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.351770    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144204e6-4a23-4dfe-9686-e6da6c450145-tls-certs\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.378607 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.378577    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-h2qzd"]
Nov 03 18:50:25.380868 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.380852    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.404192 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.404162    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-h2qzd"]
Nov 03 18:50:25.406870 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.406840    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgl9q\" (UniqueName: \"kubernetes.io/projected/144204e6-4a23-4dfe-9686-e6da6c450145-kube-api-access-lgl9q\") pod \"model-serving-api-75bd89fdfc-tx2vn\" (UID: \"144204e6-4a23-4dfe-9686-e6da6c450145\") " pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.429983 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.429959    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Nov 03 18:50:25.429983 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.429961    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-mxcl2\""
Nov 03 18:50:25.430136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.429961    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Nov 03 18:50:25.450386 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.450362    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-metrics-certs\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.450493 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.450417    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-cert\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.450493 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.450479    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg75m\" (UniqueName: \"kubernetes.io/projected/3d3b180a-7891-49c6-9656-1eb34b3da879-kube-api-access-sg75m\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.544202 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.544173    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:25.551427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.551397    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-cert\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.551549 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.551442    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg75m\" (UniqueName: \"kubernetes.io/projected/3d3b180a-7891-49c6-9656-1eb34b3da879-kube-api-access-sg75m\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.551549 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.551512    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-metrics-certs\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.553945 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.553923    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-cert\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.553995 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.553923    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3b180a-7891-49c6-9656-1eb34b3da879-metrics-certs\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.614191 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.611742    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg75m\" (UniqueName: \"kubernetes.io/projected/3d3b180a-7891-49c6-9656-1eb34b3da879-kube-api-access-sg75m\") pod \"odh-model-controller-f87fcf56c-h2qzd\" (UID: \"3d3b180a-7891-49c6-9656-1eb34b3da879\") " pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.692873 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.692838    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:25.908112 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.908035    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-tx2vn"]
Nov 03 18:50:25.914660 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:50:25.914635    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144204e6_4a23_4dfe_9686_e6da6c450145.slice/crio-d02ccc45b32eb06bb24bc8d502fa96d4943eb99d660cb81037781bc0a3b2ff9e WatchSource:0}: Error finding container d02ccc45b32eb06bb24bc8d502fa96d4943eb99d660cb81037781bc0a3b2ff9e: Status 404 returned error can't find the container with id d02ccc45b32eb06bb24bc8d502fa96d4943eb99d660cb81037781bc0a3b2ff9e
Nov 03 18:50:25.916547 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:25.916527    4944 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:50:26.055200 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:26.055168    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-h2qzd"]
Nov 03 18:50:26.062214 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:50:26.062190    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3b180a_7891_49c6_9656_1eb34b3da879.slice/crio-204ca5777f0329ca728305abda1127952f99804dca70350505db3acac11fd863 WatchSource:0}: Error finding container 204ca5777f0329ca728305abda1127952f99804dca70350505db3acac11fd863: Status 404 returned error can't find the container with id 204ca5777f0329ca728305abda1127952f99804dca70350505db3acac11fd863
Nov 03 18:50:26.128887 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:26.128855    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn" event={"ID":"144204e6-4a23-4dfe-9686-e6da6c450145","Type":"ContainerStarted","Data":"d02ccc45b32eb06bb24bc8d502fa96d4943eb99d660cb81037781bc0a3b2ff9e"}
Nov 03 18:50:26.129917 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:26.129893    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd" event={"ID":"3d3b180a-7891-49c6-9656-1eb34b3da879","Type":"ContainerStarted","Data":"204ca5777f0329ca728305abda1127952f99804dca70350505db3acac11fd863"}
Nov 03 18:50:26.474751 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:26.474712    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:29.143600 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:29.143569    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn" event={"ID":"144204e6-4a23-4dfe-9686-e6da6c450145","Type":"ContainerStarted","Data":"4d58223d52ba279a7f3eb1dae4076b176427e41496f6298b4c8042b013ddccbb"}
Nov 03 18:50:29.143989 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:29.143681    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:29.292488 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:29.292432    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn" podStartSLOduration=1.9321862140000001 podStartE2EDuration="4.292417116s" totalImagesPullingTime="2.360230902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:29.291085674 +0000 UTC m=+327.568439257" watchObservedRunningTime="2026-11-03 18:50:29.292417116 +0000 UTC m=+327.569770696"
Nov 03 18:50:30.149461 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:30.149427    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd" event={"ID":"3d3b180a-7891-49c6-9656-1eb34b3da879","Type":"ContainerStarted","Data":"acbee380272284b68ebec47b579bf486c38c4b1512af1a51ff5d92e469429f4f"}
Nov 03 18:50:30.149860 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:30.149766    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:50:30.261352 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:30.261302    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd" podStartSLOduration=2.193446443 podStartE2EDuration="5.261287838s" totalImagesPullingTime="3.067841395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:50:30.260374098 +0000 UTC m=+328.537727719" watchObservedRunningTime="2026-11-03 18:50:30.261287838 +0000 UTC m=+328.538641414"
Nov 03 18:50:31.548229 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:31.548194    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:31.789088 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:31.789056    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:32.032404 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:32.032368    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:32.275785 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:32.275748    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:32.519526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:32.519496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:32.761780 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:32.761751    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:50:35.222906 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:50:35.222865    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:50:40.153718 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:40.153688    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn"
Nov 03 18:50:42.159259 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:50:42.159232    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-h2qzd"
Nov 03 18:51:05.223596 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:51:05.223559    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:51:08.772522 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:51:08.772491    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:17.533718 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:51:17.533679    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:33.585768 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:51:33.585733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:33.835433 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:51:33.835405    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:51:35.222677 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:51:35.222634    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:52:00.894174 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:00.894142    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:03.689915 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:03.689876    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:05.222624 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:52:05.222586    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:52:05.485876 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:05.485805    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:05.747527 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:05.747468    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:06.028525 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:06.028453    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:06.489954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:06.489921    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:06.924562 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:06.924482    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:07.948058 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:07.948021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:09.694549 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:09.694515    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:10.090054 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:10.090026    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:11.891684 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.891653    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"]
Nov 03 18:52:11.894782 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.894764    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:11.940688 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.940664    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:11.953170 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.953152    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 18:52:11.999176 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.999148    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 18:52:11.999280 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:11.999190    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mxg6s\""
Nov 03 18:52:12.000359 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.000339    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"]
Nov 03 18:52:12.020426 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.020403    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljlr\" (UniqueName: \"kubernetes.io/projected/5b34af26-bee6-4b33-983a-29ef3f37657a-kube-api-access-4ljlr\") pod \"authorino-operator-68d6dd76dc-ws6kr\" (UID: \"5b34af26-bee6-4b33-983a-29ef3f37657a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:12.120858 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.120822    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljlr\" (UniqueName: \"kubernetes.io/projected/5b34af26-bee6-4b33-983a-29ef3f37657a-kube-api-access-4ljlr\") pod \"authorino-operator-68d6dd76dc-ws6kr\" (UID: \"5b34af26-bee6-4b33-983a-29ef3f37657a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:12.179152 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.179092    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljlr\" (UniqueName: \"kubernetes.io/projected/5b34af26-bee6-4b33-983a-29ef3f37657a-kube-api-access-4ljlr\") pod \"authorino-operator-68d6dd76dc-ws6kr\" (UID: \"5b34af26-bee6-4b33-983a-29ef3f37657a\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:12.194727 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.194704    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:12.207272 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.207254    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:12.582331 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:12.582300    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"]
Nov 03 18:52:12.589817 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:52:12.589772    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b34af26_bee6_4b33_983a_29ef3f37657a.slice/crio-88621e3f64046a098c50bba52356075f657bbc482558363cc63a91b50d10d65d WatchSource:0}: Error finding container 88621e3f64046a098c50bba52356075f657bbc482558363cc63a91b50d10d65d: Status 404 returned error can't find the container with id 88621e3f64046a098c50bba52356075f657bbc482558363cc63a91b50d10d65d
Nov 03 18:52:13.491925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:13.491889    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:13.528095 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:13.528059    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr" event={"ID":"5b34af26-bee6-4b33-983a-29ef3f37657a","Type":"ContainerStarted","Data":"88621e3f64046a098c50bba52356075f657bbc482558363cc63a91b50d10d65d"}
Nov 03 18:52:13.752106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:13.752029    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:15.517024 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:15.516994    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:15.766383 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:15.766348    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:16.542709 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:16.542670    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr" event={"ID":"5b34af26-bee6-4b33-983a-29ef3f37657a","Type":"ContainerStarted","Data":"cf88ecc700274a5b312e10fc51a594add32c4b0a488c0cc596f4cb3b818d4b73"}
Nov 03 18:52:16.543104 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:16.542775    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:16.650644 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:16.650541    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr" podStartSLOduration=1.886056707 podStartE2EDuration="5.650524091s" totalImagesPullingTime="3.764467384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:52:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:52:16.650416585 +0000 UTC m=+434.927770169" watchObservedRunningTime="2026-11-03 18:52:16.650524091 +0000 UTC m=+434.927877678"
Nov 03 18:52:17.877995 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:17.877959    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:21.309246 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:21.309213    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:21.573066 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:21.572982    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:23.814989 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:23.814953    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:27.549048 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:27.549012    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-ws6kr"
Nov 03 18:52:30.280986 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:30.280944    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:30.563607 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:30.563527    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:32.745734 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:32.745699    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:32.995847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:32.995764    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:35.222871 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:52:35.222828    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:52:35.529109 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:35.529062    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:35.779268 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:35.779176    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:40.287887 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:40.287848    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:40.541198 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:40.541115    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:40.802706 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:40.802618    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:49.160534 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:49.160496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:49.404897 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:49.404870    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:56.595681 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:56.595643    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:52:59.581666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:52:59.581637    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:03.170721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:03.170690    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:05.222650 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:53:05.222611    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:53:18.226670 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:18.226586    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:24.446226 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:24.446191    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:26.385608 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:26.385577    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:29.828713 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:29.828680    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:30.177753 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:30.177677    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:30.424723 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:30.424691    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:31.280463 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.280425    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:31.283822 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.283778    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:31.335634 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.335601    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q5hf9\""
Nov 03 18:53:31.336893 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.336866    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:31.350324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.350302    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvkh\" (UniqueName: \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\") pod \"authorino-57f7f8659f-4ksm7\" (UID: \"68990b44-2340-44cd-9e2e-5dae7a74f65f\") " pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:31.450959 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.450931    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvkh\" (UniqueName: \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\") pod \"authorino-57f7f8659f-4ksm7\" (UID: \"68990b44-2340-44cd-9e2e-5dae7a74f65f\") " pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:31.508594 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.508558    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvkh\" (UniqueName: \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\") pod \"authorino-57f7f8659f-4ksm7\" (UID: \"68990b44-2340-44cd-9e2e-5dae7a74f65f\") " pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:31.549169 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.549080    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:31.595200 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.595174    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:31.798341 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.798309    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:31.955613 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:31.955537    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:31.963050 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:53:31.963024    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68990b44_2340_44cd_9e2e_5dae7a74f65f.slice/crio-01210abf18e2ca1ddea0cf1789a5bb93971f27db16a12dd78609c6617b806cfb WatchSource:0}: Error finding container 01210abf18e2ca1ddea0cf1789a5bb93971f27db16a12dd78609c6617b806cfb: Status 404 returned error can't find the container with id 01210abf18e2ca1ddea0cf1789a5bb93971f27db16a12dd78609c6617b806cfb
Nov 03 18:53:32.836423 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:32.836390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" event={"ID":"68990b44-2340-44cd-9e2e-5dae7a74f65f","Type":"ContainerStarted","Data":"01210abf18e2ca1ddea0cf1789a5bb93971f27db16a12dd78609c6617b806cfb"}
Nov 03 18:53:33.963782 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:33.963746    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:53:33.967060 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:33.967037    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:53:34.019501 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.019466    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:53:34.071118 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.071085    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqfjh\" (UniqueName: \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\") pod \"authorino-56c6c9ddd5-zvp24\" (UID: \"f6744eb3-2b9f-4f82-a40f-6848c97758c5\") " pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:53:34.171646 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.171603    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqfjh\" (UniqueName: \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\") pod \"authorino-56c6c9ddd5-zvp24\" (UID: \"f6744eb3-2b9f-4f82-a40f-6848c97758c5\") " pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:53:34.229640 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.229572    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqfjh\" (UniqueName: \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\") pod \"authorino-56c6c9ddd5-zvp24\" (UID: \"f6744eb3-2b9f-4f82-a40f-6848c97758c5\") " pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:53:34.280492 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.280457    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:53:34.655506 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.655470    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:53:34.663587 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:53:34.663545    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6744eb3_2b9f_4f82_a40f_6848c97758c5.slice/crio-bc5daa0d3ac40786d2846a860729f0a93e562b01da629269d0b68222c8587b4b WatchSource:0}: Error finding container bc5daa0d3ac40786d2846a860729f0a93e562b01da629269d0b68222c8587b4b: Status 404 returned error can't find the container with id bc5daa0d3ac40786d2846a860729f0a93e562b01da629269d0b68222c8587b4b
Nov 03 18:53:34.846136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:34.846088    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" event={"ID":"f6744eb3-2b9f-4f82-a40f-6848c97758c5","Type":"ContainerStarted","Data":"bc5daa0d3ac40786d2846a860729f0a93e562b01da629269d0b68222c8587b4b"}
Nov 03 18:53:35.224706 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:53:35.224671    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:53:35.851223 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:35.851188    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" event={"ID":"68990b44-2340-44cd-9e2e-5dae7a74f65f","Type":"ContainerStarted","Data":"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"}
Nov 03 18:53:35.957603 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:35.957552    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" podStartSLOduration=1.9086211039999998 podStartE2EDuration="4.957540225s" totalImagesPullingTime="3.048919121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:35.95712326 +0000 UTC m=+514.234476843" watchObservedRunningTime="2026-11-03 18:53:35.957540225 +0000 UTC m=+514.234893809"
Nov 03 18:53:36.855803 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:36.855757    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" event={"ID":"f6744eb3-2b9f-4f82-a40f-6848c97758c5","Type":"ContainerStarted","Data":"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"}
Nov 03 18:53:36.962517 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:36.962453    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" podStartSLOduration=2.483145078 podStartE2EDuration="3.962435652s" totalImagesPullingTime="1.479290574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:53:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:53:36.962081074 +0000 UTC m=+515.239434658" watchObservedRunningTime="2026-11-03 18:53:36.962435652 +0000 UTC m=+515.239789236"
Nov 03 18:53:37.009615 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:37.009582    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:37.859583 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:37.859544    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" podUID="68990b44-2340-44cd-9e2e-5dae7a74f65f" containerName="authorino" containerID="cri-o://2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e" gracePeriod=30
Nov 03 18:53:38.114087 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.114029    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:38.208476 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.208443    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\" (UniqueName: \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\") pod \"68990b44-2340-44cd-9e2e-5dae7a74f65f\" (UID: \"68990b44-2340-44cd-9e2e-5dae7a74f65f\") "
Nov 03 18:53:38.210648 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.210621    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh" pod "68990b44-2340-44cd-9e2e-5dae7a74f65f" (UID: "68990b44-2340-44cd-9e2e-5dae7a74f65f"). InnerVolumeSpecName "kube-api-access-ttvkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:53:38.309913 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.309882    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttvkh\" (UniqueName: \"kubernetes.io/projected/68990b44-2340-44cd-9e2e-5dae7a74f65f-kube-api-access-ttvkh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:53:38.865218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.865183    4944 generic.go:356] "Generic (PLEG): container finished" podID="68990b44-2340-44cd-9e2e-5dae7a74f65f" containerID="2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e" exitCode=0
Nov 03 18:53:38.865625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.865235    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-4ksm7"
Nov 03 18:53:38.865625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.865258    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" event={"ID":"68990b44-2340-44cd-9e2e-5dae7a74f65f","Type":"ContainerDied","Data":"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"}
Nov 03 18:53:38.865625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.865290    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-4ksm7" event={"ID":"68990b44-2340-44cd-9e2e-5dae7a74f65f","Type":"ContainerDied","Data":"01210abf18e2ca1ddea0cf1789a5bb93971f27db16a12dd78609c6617b806cfb"}
Nov 03 18:53:38.865625 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.865306    4944 scope.go:122] "RemoveContainer" containerID="2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"
Nov 03 18:53:38.873166 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.873150    4944 scope.go:122] "RemoveContainer" containerID="2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"
Nov 03 18:53:38.873436 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:53:38.873416    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e\": container with ID starting with 2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e not found: ID does not exist" containerID="2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"
Nov 03 18:53:38.873505 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:38.873444    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e"} err="failed to get container status \"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e\": rpc error: code = NotFound desc = could not find container \"2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e\": container with ID starting with 2e8b88f03475e2d7f8c59528d4c04056bf04e7485a7ac817dfff365b8d40dc8e not found: ID does not exist"
Nov 03 18:53:39.016171 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:39.016139    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:39.062469 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:39.062426    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-4ksm7"]
Nov 03 18:53:39.931649 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:39.931614    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68990b44-2340-44cd-9e2e-5dae7a74f65f" path="/var/lib/kubelet/pods/68990b44-2340-44cd-9e2e-5dae7a74f65f/volumes"
Nov 03 18:53:43.979334 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:43.979297    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:51.273647 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:51.273611    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:59.544518 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:59.544488    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:53:59.792573 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:53:59.792536    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:00.317023 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:00.316990    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:05.222390 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:54:05.222355    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:54:16.502059 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.502019    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:16.696528 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.696491    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:16.696849 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.696835    4944 cpu_manager.go:414] "containerMap: removing container" podUID="68990b44-2340-44cd-9e2e-5dae7a74f65f" containerName="authorino"
Nov 03 18:54:16.696914 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.696850    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68990b44-2340-44cd-9e2e-5dae7a74f65f" containerName="authorino"
Nov 03 18:54:16.696952 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.696913    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="68990b44-2340-44cd-9e2e-5dae7a74f65f" containerName="authorino"
Nov 03 18:54:16.698893 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.698876    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:16.718370 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.718345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tgw\" (UniqueName: \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\") pod \"authorino-6f6944bcdd-pxkkd\" (UID: \"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6\") " pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:16.750992 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.750967    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:16.818867 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.818765    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tgw\" (UniqueName: \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\") pod \"authorino-6f6944bcdd-pxkkd\" (UID: \"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6\") " pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:16.835188 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.835162    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:16.876203 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:16.876173    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tgw\" (UniqueName: \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\") pod \"authorino-6f6944bcdd-pxkkd\" (UID: \"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6\") " pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:17.009388 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.009353    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:17.033471 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.033439    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:17.063022 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.062993    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:17.065983 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.065960    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:17.117681 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.117635    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:17.121124 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.121090    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmkb\" (UniqueName: \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\") pod \"authorino-5f654d8447-2w9jr\" (UID: \"82a8ea4c-a241-44fb-88c5-d558c5f08863\") " pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:17.221781 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.221745    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmkb\" (UniqueName: \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\") pod \"authorino-5f654d8447-2w9jr\" (UID: \"82a8ea4c-a241-44fb-88c5-d558c5f08863\") " pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:17.279241 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.279212    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmkb\" (UniqueName: \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\") pod \"authorino-5f654d8447-2w9jr\" (UID: \"82a8ea4c-a241-44fb-88c5-d558c5f08863\") " pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:17.372154 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.372069    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:17.379083 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:54:17.379053    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a5e607_dd2b_4ca2_bc5b_5f715cd36da6.slice/crio-98633eaf30ac07f6a0668c59c475e6487e4899cd9e1e05bad1cb9797db4c9442 WatchSource:0}: Error finding container 98633eaf30ac07f6a0668c59c475e6487e4899cd9e1e05bad1cb9797db4c9442: Status 404 returned error can't find the container with id 98633eaf30ac07f6a0668c59c475e6487e4899cd9e1e05bad1cb9797db4c9442
Nov 03 18:54:17.380020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.379999    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:17.531770 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.531736    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:17.560898 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.560863    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:54:17.564248 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.564231    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.613300 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.613274    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 18:54:17.617731 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.617710    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:54:17.624896 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.624839    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbgv\" (UniqueName: \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.624896 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.624873    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.726111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.726079    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbgv\" (UniqueName: \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.726111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.726110    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.728412 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.728389    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.742473 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.742449    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:17.749613 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:54:17.749590    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82a8ea4c_a241_44fb_88c5_d558c5f08863.slice/crio-9ed3502074a6aabf332814e886284616ecaa0f7ae8c30f22d8e6024679015593 WatchSource:0}: Error finding container 9ed3502074a6aabf332814e886284616ecaa0f7ae8c30f22d8e6024679015593: Status 404 returned error can't find the container with id 9ed3502074a6aabf332814e886284616ecaa0f7ae8c30f22d8e6024679015593
Nov 03 18:54:17.783168 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.783141    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbgv\" (UniqueName: \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\") pod \"authorino-d69bd8d6f-k8f4v\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") " pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:17.874961 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:17.874909    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:54:18.019088 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:18.019048    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f654d8447-2w9jr" event={"ID":"82a8ea4c-a241-44fb-88c5-d558c5f08863","Type":"ContainerStarted","Data":"9ed3502074a6aabf332814e886284616ecaa0f7ae8c30f22d8e6024679015593"}
Nov 03 18:54:18.020208 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:18.020189    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" event={"ID":"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6","Type":"ContainerStarted","Data":"98633eaf30ac07f6a0668c59c475e6487e4899cd9e1e05bad1cb9797db4c9442"}
Nov 03 18:54:18.242721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:18.242691    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:54:18.250127 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:54:18.250089    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddcc69b60_b4e5_4427_a0c1_d8b352158343.slice/crio-2445867cbf2f470b87d87f85b1f241198554916a2d2a9e45ecc3e59568d534a8 WatchSource:0}: Error finding container 2445867cbf2f470b87d87f85b1f241198554916a2d2a9e45ecc3e59568d534a8: Status 404 returned error can't find the container with id 2445867cbf2f470b87d87f85b1f241198554916a2d2a9e45ecc3e59568d534a8
Nov 03 18:54:18.572334 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:18.572254    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:19.025087 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.025050    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" event={"ID":"dcc69b60-b4e5-4427-a0c1-d8b352158343","Type":"ContainerStarted","Data":"2445867cbf2f470b87d87f85b1f241198554916a2d2a9e45ecc3e59568d534a8"}
Nov 03 18:54:19.026411 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.026387    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" event={"ID":"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6","Type":"ContainerStarted","Data":"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"}
Nov 03 18:54:19.026492 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.026464    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" podUID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" containerName="authorino" containerID="cri-o://eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329" gracePeriod=30
Nov 03 18:54:19.133961 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.133917    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" podStartSLOduration=1.6710188750000001 podStartE2EDuration="3.133905078s" totalImagesPullingTime="1.462886203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:19.132276128 +0000 UTC m=+557.409629714" watchObservedRunningTime="2026-11-03 18:54:19.133905078 +0000 UTC m=+557.411258683"
Nov 03 18:54:19.281996 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.281972    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:19.344090 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.344030    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\" (UniqueName: \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\") pod \"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6\" (UID: \"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6\") "
Nov 03 18:54:19.346111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.346086    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw" pod "f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" (UID: "f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6"). InnerVolumeSpecName "kube-api-access-f5tgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:54:19.444918 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:19.444753    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5tgw\" (UniqueName: \"kubernetes.io/projected/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6-kube-api-access-f5tgw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:20.031510 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.031479    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" event={"ID":"dcc69b60-b4e5-4427-a0c1-d8b352158343","Type":"ContainerStarted","Data":"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"}
Nov 03 18:54:20.033083 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.033051    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f654d8447-2w9jr" event={"ID":"82a8ea4c-a241-44fb-88c5-d558c5f08863","Type":"ContainerStarted","Data":"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"}
Nov 03 18:54:20.033233 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.033104    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f654d8447-2w9jr" podUID="82a8ea4c-a241-44fb-88c5-d558c5f08863" containerName="authorino" containerID="cri-o://6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd" gracePeriod=30
Nov 03 18:54:20.034207 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.034187    4944 generic.go:356] "Generic (PLEG): container finished" podID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" containerID="eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329" exitCode=0
Nov 03 18:54:20.034303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.034219    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" event={"ID":"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6","Type":"ContainerDied","Data":"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"}
Nov 03 18:54:20.034303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.034268    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd" event={"ID":"f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6","Type":"ContainerDied","Data":"98633eaf30ac07f6a0668c59c475e6487e4899cd9e1e05bad1cb9797db4c9442"}
Nov 03 18:54:20.034303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.034293    4944 scope.go:122] "RemoveContainer" containerID="eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"
Nov 03 18:54:20.034303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.034226    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-pxkkd"
Nov 03 18:54:20.042681 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.042662    4944 scope.go:122] "RemoveContainer" containerID="eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"
Nov 03 18:54:20.042981 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:54:20.042960    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329\": container with ID starting with eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329 not found: ID does not exist" containerID="eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"
Nov 03 18:54:20.043083 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.042988    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329"} err="failed to get container status \"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329\": rpc error: code = NotFound desc = could not find container \"eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329\": container with ID starting with eca718e08643fc3e2215c7f8e2fd8f851200c95c5b7fa296cb245a7b57c77329 not found: ID does not exist"
Nov 03 18:54:20.138954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.138904    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" podStartSLOduration=1.945499523 podStartE2EDuration="3.138885853s" totalImagesPullingTime="1.19338633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:20.138171704 +0000 UTC m=+558.415525287" watchObservedRunningTime="2026-11-03 18:54:20.138885853 +0000 UTC m=+558.416239438"
Nov 03 18:54:20.162711 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.162679    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:54:20.162889 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.162870    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" podUID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" containerName="authorino" containerID="cri-o://5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d" gracePeriod=30
Nov 03 18:54:20.247452 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.247399    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f654d8447-2w9jr" podStartSLOduration=1.914270162 podStartE2EDuration="3.247382682s" totalImagesPullingTime="1.33311252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:54:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:54:20.246287825 +0000 UTC m=+558.523641407" watchObservedRunningTime="2026-11-03 18:54:20.247382682 +0000 UTC m=+558.524736266"
Nov 03 18:54:20.284495 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.284473    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:20.349548 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.349513    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:20.352811 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.352757    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\" (UniqueName: \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\") pod \"82a8ea4c-a241-44fb-88c5-d558c5f08863\" (UID: \"82a8ea4c-a241-44fb-88c5-d558c5f08863\") "
Nov 03 18:54:20.355530 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.355498    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb" pod "82a8ea4c-a241-44fb-88c5-d558c5f08863" (UID: "82a8ea4c-a241-44fb-88c5-d558c5f08863"). InnerVolumeSpecName "kube-api-access-swmkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:54:20.355620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.355570    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-pxkkd"]
Nov 03 18:54:20.388262 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.388238    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:54:20.454186 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.454154    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\" (UniqueName: \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\") pod \"f6744eb3-2b9f-4f82-a40f-6848c97758c5\" (UID: \"f6744eb3-2b9f-4f82-a40f-6848c97758c5\") "
Nov 03 18:54:20.454351 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.454336    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmkb\" (UniqueName: \"kubernetes.io/projected/82a8ea4c-a241-44fb-88c5-d558c5f08863-kube-api-access-swmkb\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:20.456161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.456135    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh" pod "f6744eb3-2b9f-4f82-a40f-6848c97758c5" (UID: "f6744eb3-2b9f-4f82-a40f-6848c97758c5"). InnerVolumeSpecName "kube-api-access-pqfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:54:20.555577 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:20.555551    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqfjh\" (UniqueName: \"kubernetes.io/projected/f6744eb3-2b9f-4f82-a40f-6848c97758c5-kube-api-access-pqfjh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:54:21.040079 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.040043    4944 generic.go:356] "Generic (PLEG): container finished" podID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" containerID="5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d" exitCode=0
Nov 03 18:54:21.040526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.040086    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24"
Nov 03 18:54:21.040526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.040131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" event={"ID":"f6744eb3-2b9f-4f82-a40f-6848c97758c5","Type":"ContainerDied","Data":"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"}
Nov 03 18:54:21.040526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.040168    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-zvp24" event={"ID":"f6744eb3-2b9f-4f82-a40f-6848c97758c5","Type":"ContainerDied","Data":"bc5daa0d3ac40786d2846a860729f0a93e562b01da629269d0b68222c8587b4b"}
Nov 03 18:54:21.040526 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.040187    4944 scope.go:122] "RemoveContainer" containerID="5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"
Nov 03 18:54:21.041480 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.041457    4944 generic.go:356] "Generic (PLEG): container finished" podID="82a8ea4c-a241-44fb-88c5-d558c5f08863" containerID="6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd" exitCode=0
Nov 03 18:54:21.041590 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.041483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f654d8447-2w9jr" event={"ID":"82a8ea4c-a241-44fb-88c5-d558c5f08863","Type":"ContainerDied","Data":"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"}
Nov 03 18:54:21.041590 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.041495    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f654d8447-2w9jr"
Nov 03 18:54:21.041590 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.041521    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f654d8447-2w9jr" event={"ID":"82a8ea4c-a241-44fb-88c5-d558c5f08863","Type":"ContainerDied","Data":"9ed3502074a6aabf332814e886284616ecaa0f7ae8c30f22d8e6024679015593"}
Nov 03 18:54:21.049075 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.048806    4944 scope.go:122] "RemoveContainer" containerID="5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"
Nov 03 18:54:21.049150 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:54:21.049134    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d\": container with ID starting with 5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d not found: ID does not exist" containerID="5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"
Nov 03 18:54:21.049205 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.049162    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d"} err="failed to get container status \"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d\": rpc error: code = NotFound desc = could not find container \"5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d\": container with ID starting with 5e465ccced62f1af5889f27bbe1b577888590e13ec455b1cab08e828ad3e688d not found: ID does not exist"
Nov 03 18:54:21.049205 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.049184    4944 scope.go:122] "RemoveContainer" containerID="6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"
Nov 03 18:54:21.056667 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.056647    4944 scope.go:122] "RemoveContainer" containerID="6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"
Nov 03 18:54:21.056939 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:54:21.056922    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd\": container with ID starting with 6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd not found: ID does not exist" containerID="6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"
Nov 03 18:54:21.057003 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.056945    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd"} err="failed to get container status \"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd\": rpc error: code = NotFound desc = could not find container \"6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd\": container with ID starting with 6b6cf10a135355a477309893180b477bf6e9ee652520da4871ea894b7a2514dd not found: ID does not exist"
Nov 03 18:54:21.192301 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.192272    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:21.201205 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.201180    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f654d8447-2w9jr"]
Nov 03 18:54:21.305473 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.305405    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:54:21.309707 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.309685    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-zvp24"]
Nov 03 18:54:21.932019 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.931987    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82a8ea4c-a241-44fb-88c5-d558c5f08863" path="/var/lib/kubelet/pods/82a8ea4c-a241-44fb-88c5-d558c5f08863/volumes"
Nov 03 18:54:21.932320 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.932306    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" path="/var/lib/kubelet/pods/f6744eb3-2b9f-4f82-a40f-6848c97758c5/volumes"
Nov 03 18:54:21.932591 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:21.932580    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" path="/var/lib/kubelet/pods/f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6/volumes"
Nov 03 18:54:22.594919 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:22.594876    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:22.839575 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:22.839536    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:27.127184 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:27.127147    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:27.384377 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:27.384291    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:27.657214 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:27.657137    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:33.877765 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:33.877734    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:34.134102 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:34.134027    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:35.223778 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:54:35.223738    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:54:38.867383 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:38.867345    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:54:39.121894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:54:39.121785    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:01.974665 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:01.974639    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:55:01.975915 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:01.975893    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 18:55:02.957990 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:02.957953    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:03.258614 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:03.258581    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:04.883085 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:04.883051    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:05.124377 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:05.124347    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:05.223907 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:55:05.223833    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:55:05.369639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:05.369605    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:16.310677 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:16.310640    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:16.557394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:16.557359    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:18.901088 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:18.901053    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:35.223064 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:55:35.223028    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:55:53.448901 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:53.448863    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:55:53.702166 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:55:53.702092    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:05.223302 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:56:05.223269    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:56:09.833131 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833097    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"]
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833404    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" containerName="authorino"
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833416    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" containerName="authorino"
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833428    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" containerName="authorino"
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833433    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" containerName="authorino"
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833447    4944 cpu_manager.go:414] "containerMap: removing container" podUID="82a8ea4c-a241-44fb-88c5-d558c5f08863" containerName="authorino"
Nov 03 18:56:09.833520 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833470    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82a8ea4c-a241-44fb-88c5-d558c5f08863" containerName="authorino"
Nov 03 18:56:09.833702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833531    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6744eb3-2b9f-4f82-a40f-6848c97758c5" containerName="authorino"
Nov 03 18:56:09.833702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833541    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a5e607-dd2b-4ca2-bc5b-5f715cd36da6" containerName="authorino"
Nov 03 18:56:09.833702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.833547    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="82a8ea4c-a241-44fb-88c5-d558c5f08863" containerName="authorino"
Nov 03 18:56:09.835587 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.835573    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:09.884376 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.884352    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 18:56:09.930294 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.930269    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z8pj6\""
Nov 03 18:56:09.930582 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.930565    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 18:56:09.930674 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.930594    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 18:56:09.932327 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:09.932309    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"]
Nov 03 18:56:10.031442 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031408    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.031589 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031449    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.031589 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031553    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.031675 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031612    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.031675 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031649    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.031760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.031701    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjpq\" (UniqueName: \"kubernetes.io/projected/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-kube-api-access-qvjpq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.039557 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.039531    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:10.132184 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132105    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132184 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132142    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132184 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132171    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjpq\" (UniqueName: \"kubernetes.io/projected/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-kube-api-access-qvjpq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132218    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132246    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132308    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132600    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132701 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132654    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.132701 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.132674    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.134424 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.134403    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.134672 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.134642    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.189426 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.189402    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjpq\" (UniqueName: \"kubernetes.io/projected/0b0bd44e-1b16-4195-9d46-c3a59e9a5534-kube-api-access-qvjpq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr\" (UID: \"0b0bd44e-1b16-4195-9d46-c3a59e9a5534\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.446701 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.446623    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:10.810751 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.810710    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"]
Nov 03 18:56:10.817181 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:56:10.817157    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0bd44e_1b16_4195_9d46_c3a59e9a5534.slice/crio-8675c99d110aca3387d3786e92b594c25eb6f34cd46c2f192d474c2b3cc1675e WatchSource:0}: Error finding container 8675c99d110aca3387d3786e92b594c25eb6f34cd46c2f192d474c2b3cc1675e: Status 404 returned error can't find the container with id 8675c99d110aca3387d3786e92b594c25eb6f34cd46c2f192d474c2b3cc1675e
Nov 03 18:56:10.818432 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:10.818415    4944 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 18:56:11.473979 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:11.473943    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr" event={"ID":"0b0bd44e-1b16-4195-9d46-c3a59e9a5534","Type":"ContainerStarted","Data":"8675c99d110aca3387d3786e92b594c25eb6f34cd46c2f192d474c2b3cc1675e"}
Nov 03 18:56:13.484738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:13.484649    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr" event={"ID":"0b0bd44e-1b16-4195-9d46-c3a59e9a5534","Type":"ContainerStarted","Data":"98f98c32351ba229cc529a2910a6537304edc47123da498e0c8740d0f83fe551"}
Nov 03 18:56:13.594135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:13.594072    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr" podStartSLOduration=2.353547394 podStartE2EDuration="4.594051338s" totalImagesPullingTime="2.240503944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:13.593551023 +0000 UTC m=+671.870904611" watchObservedRunningTime="2026-11-03 18:56:13.594051338 +0000 UTC m=+671.871404922"
Nov 03 18:56:14.446974 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:14.446934    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:15.604857 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.604823    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"]
Nov 03 18:56:15.607226 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.607207    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678228 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678193    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678228 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678226    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678417 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678246    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678417 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678370    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1176c49-737c-4882-905e-799b99aa88fb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678417 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678413    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.678549 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.678431    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rrr\" (UniqueName: \"kubernetes.io/projected/e1176c49-737c-4882-905e-799b99aa88fb-kube-api-access-q6rrr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.688959 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.688934    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 18:56:15.704965 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.704934    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"]
Nov 03 18:56:15.779207 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779177    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779211    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779243    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1176c49-737c-4882-905e-799b99aa88fb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779282    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779303    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rrr\" (UniqueName: \"kubernetes.io/projected/e1176c49-737c-4882-905e-799b99aa88fb-kube-api-access-q6rrr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779326    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779657 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779633    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779649    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.779763 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.779733    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.781556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.781536    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1176c49-737c-4882-905e-799b99aa88fb-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.781697 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.781681    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1176c49-737c-4882-905e-799b99aa88fb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.812177 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.812145    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:15.836110 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.836082    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rrr\" (UniqueName: \"kubernetes.io/projected/e1176c49-737c-4882-905e-799b99aa88fb-kube-api-access-q6rrr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7\" (UID: \"e1176c49-737c-4882-905e-799b99aa88fb\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:15.918110 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:15.918029    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:16.285966 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:16.285936    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"]
Nov 03 18:56:16.292407 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:56:16.292376    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1176c49_737c_4882_905e_799b99aa88fb.slice/crio-6aa1675503c79cf161846b38a2dd3466af835d6fdb5fe3b1aabf2dae848d15bc WatchSource:0}: Error finding container 6aa1675503c79cf161846b38a2dd3466af835d6fdb5fe3b1aabf2dae848d15bc: Status 404 returned error can't find the container with id 6aa1675503c79cf161846b38a2dd3466af835d6fdb5fe3b1aabf2dae848d15bc
Nov 03 18:56:16.498205 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:16.498175    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" event={"ID":"e1176c49-737c-4882-905e-799b99aa88fb","Type":"ContainerStarted","Data":"6aa1675503c79cf161846b38a2dd3466af835d6fdb5fe3b1aabf2dae848d15bc"}
Nov 03 18:56:17.505188 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:17.505148    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" event={"ID":"e1176c49-737c-4882-905e-799b99aa88fb","Type":"ContainerStarted","Data":"be826a8b9d4f0fca524efbf4833fdcf88bb856b333a75eedc6d159effc883eab"}
Nov 03 18:56:17.614860 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:17.614783    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podStartSLOduration=2.028984824 podStartE2EDuration="2.614769323s" totalImagesPullingTime="585.784499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:17.614125007 +0000 UTC m=+675.891478602" watchObservedRunningTime="2026-11-03 18:56:17.614769323 +0000 UTC m=+675.892122897"
Nov 03 18:56:17.918504 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:17.918412    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:19.244864 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:19.244835    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:20.447120 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:20.447081    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:20.459518 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:20.459495    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:20.527439 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:20.527413    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr"
Nov 03 18:56:22.651815 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:22.651759    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:25.918827 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:25.918779    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:25.936202 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:25.936175    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:26.551579 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:26.551551    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 18:56:27.083479 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:27.083447    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:31.079655 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.079620    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"]
Nov 03 18:56:31.082756 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.082737    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.177472 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.177432    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 18:56:31.178849 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.178830    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"]
Nov 03 18:56:31.206804 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.206759    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpxx\" (UniqueName: \"kubernetes.io/projected/dd152083-a780-4dcf-8d62-884d530b281d-kube-api-access-9bpxx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.206947 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.206824    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.207012 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.206992    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.207068 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.207050    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.207124 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.207096    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd152083-a780-4dcf-8d62-884d530b281d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.207183 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.207171    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.271867 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.271832    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:31.308383 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308345    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308543 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308412    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308543 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308441    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd152083-a780-4dcf-8d62-884d530b281d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308543 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308479    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308543 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308525    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpxx\" (UniqueName: \"kubernetes.io/projected/dd152083-a780-4dcf-8d62-884d530b281d-kube-api-access-9bpxx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308742 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308567    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.308847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308820    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.309009 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308861    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.309009 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.308893    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.310858 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.310830    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd152083-a780-4dcf-8d62-884d530b281d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.310954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.310935    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd152083-a780-4dcf-8d62-884d530b281d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.366218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.366148    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpxx\" (UniqueName: \"kubernetes.io/projected/dd152083-a780-4dcf-8d62-884d530b281d-kube-api-access-9bpxx\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-8nk8p\" (UID: \"dd152083-a780-4dcf-8d62-884d530b281d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.395059 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.395028    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:31.771892 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:31.771858    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"]
Nov 03 18:56:31.779085 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:56:31.779055    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd152083_a780_4dcf_8d62_884d530b281d.slice/crio-85a022a6181a446c9febf2c4e681bdea5f4aca95e5508858dd108ec527e27d0d WatchSource:0}: Error finding container 85a022a6181a446c9febf2c4e681bdea5f4aca95e5508858dd108ec527e27d0d: Status 404 returned error can't find the container with id 85a022a6181a446c9febf2c4e681bdea5f4aca95e5508858dd108ec527e27d0d
Nov 03 18:56:32.568915 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:32.568879    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p" event={"ID":"dd152083-a780-4dcf-8d62-884d530b281d","Type":"ContainerStarted","Data":"1f2332dae21290cdc622f36991a98bd32447783cbccbb020f56468d9b741b826"}
Nov 03 18:56:32.569279 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:32.568930    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p" event={"ID":"dd152083-a780-4dcf-8d62-884d530b281d","Type":"ContainerStarted","Data":"85a022a6181a446c9febf2c4e681bdea5f4aca95e5508858dd108ec527e27d0d"}
Nov 03 18:56:32.678719 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:32.678615    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p" podStartSLOduration=1.110545611 podStartE2EDuration="1.678595744s" totalImagesPullingTime="568.050133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:32.678075209 +0000 UTC m=+690.955428793" watchObservedRunningTime="2026-11-03 18:56:32.678595744 +0000 UTC m=+690.955949330"
Nov 03 18:56:33.395212 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:33.395179    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:34.502308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:34.502262    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:35.223319 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:56:35.223280    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:56:38.763507 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.763477    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"]
Nov 03 18:56:38.768416 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.768383    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.831494 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.831465    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 18:56:38.842068 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.842041    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"]
Nov 03 18:56:38.880078 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880047    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.880239 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880089    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.880239 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880107    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.880239 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880196    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.880384 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880248    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.880384 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.880310    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmlk\" (UniqueName: \"kubernetes.io/projected/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-kube-api-access-lxmlk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.980963 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.980929    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.980972    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981135 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.980988    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981256 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981193    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981310 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981257    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981361 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981332    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmlk\" (UniqueName: \"kubernetes.io/projected/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-kube-api-access-lxmlk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981361 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981339    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981474 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981389    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.981536 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.981516    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.983301 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.983273    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:38.983490 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:38.983475    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:39.009589 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:39.009559    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:39.038652 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:39.038593    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmlk\" (UniqueName: \"kubernetes.io/projected/3ef9f491-f8de-46a5-b4c4-de5981ca5ff2-kube-api-access-lxmlk\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn\" (UID: \"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:39.082004 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:39.081982    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:39.456674 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:39.456596    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"]
Nov 03 18:56:39.463219 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:56:39.463193    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef9f491_f8de_46a5_b4c4_de5981ca5ff2.slice/crio-7a8e2e38254155c650f8bdd0973dc8186a2f523f27708c780815b8a0a5d6f34e WatchSource:0}: Error finding container 7a8e2e38254155c650f8bdd0973dc8186a2f523f27708c780815b8a0a5d6f34e: Status 404 returned error can't find the container with id 7a8e2e38254155c650f8bdd0973dc8186a2f523f27708c780815b8a0a5d6f34e
Nov 03 18:56:39.600392 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:39.600357    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn" event={"ID":"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2","Type":"ContainerStarted","Data":"7a8e2e38254155c650f8bdd0973dc8186a2f523f27708c780815b8a0a5d6f34e"}
Nov 03 18:56:40.606015 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:40.605982    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn" event={"ID":"3ef9f491-f8de-46a5-b4c4-de5981ca5ff2","Type":"ContainerStarted","Data":"15c468ed3e912148a8e330bbd2175f0cf2b1ed7ca6dcec2669cc73a5b3fa10c2"}
Nov 03 18:56:40.716739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:40.716694    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn" podStartSLOduration=2.079218624 podStartE2EDuration="2.716679396s" totalImagesPullingTime="637.460772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:40.715811356 +0000 UTC m=+698.993164937" watchObservedRunningTime="2026-11-03 18:56:40.716679396 +0000 UTC m=+698.994032978"
Nov 03 18:56:41.082960 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.082924    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:41.254172 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.254139    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 18:56:41.257589 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.257570    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.306316 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.306290    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-26cvs\""
Nov 03 18:56:41.306316 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.306290    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 18:56:41.354773 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.354701    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 18:56:41.395980 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.395950    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:41.403599 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403564    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmtr\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403647    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403672    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403734    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403769    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403885    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403936    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.403976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.403967    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.404128 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.404072    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.408891 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.408870    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:41.504663 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504636    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmtr\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.504861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504676    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.504861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504707    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.504861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504749    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.504861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504775    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.504861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.504825    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505166 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505011    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505166 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505057    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505166 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505141    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505325 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505252    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505325 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505272    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505325 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505301    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505516 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505486    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.505853 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.505833    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.507309 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.507291    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.507463 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.507443    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.561700 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.561672    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.561859 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.561726    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmtr\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\") pod \"maas-default-gateway-openshift-default-77f556d647-7jmdj\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.569572 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.569553    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:41.626450 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.626424    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-8nk8p"
Nov 03 18:56:41.988777 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:41.988702    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 18:56:41.994345 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:56:41.994317    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5edde6c1_e803_46f8_aa0a_689f4a2f7838.slice/crio-5509ca00d76a50fd7d86084167a03b1cecce41907a3f219aa2de6616fdc34646 WatchSource:0}: Error finding container 5509ca00d76a50fd7d86084167a03b1cecce41907a3f219aa2de6616fdc34646: Status 404 returned error can't find the container with id 5509ca00d76a50fd7d86084167a03b1cecce41907a3f219aa2de6616fdc34646
Nov 03 18:56:42.616459 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:42.616419    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" event={"ID":"5edde6c1-e803-46f8-aa0a-689f4a2f7838","Type":"ContainerStarted","Data":"5509ca00d76a50fd7d86084167a03b1cecce41907a3f219aa2de6616fdc34646"}
Nov 03 18:56:48.005270 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:48.005240    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:56:48.005516 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:48.005308    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:56:48.005516 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:48.005337    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:56:48.645401 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:48.645368    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" event={"ID":"5edde6c1-e803-46f8-aa0a-689f4a2f7838","Type":"ContainerStarted","Data":"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"}
Nov 03 18:56:48.804691 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:48.804639    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podStartSLOduration=1.7964772770000002 podStartE2EDuration="7.804626642s" totalImagesPullingTime="6.008149365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:56:48.804204031 +0000 UTC m=+707.081557705" watchObservedRunningTime="2026-11-03 18:56:48.804626642 +0000 UTC m=+707.081980225"
Nov 03 18:56:49.082556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.082513    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:49.095096 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.095073    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:49.570357 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.570326    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:49.571743 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.571715    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:49.571885 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.571774    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:49.661016 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:49.660983    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn"
Nov 03 18:56:50.569940 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:50.569906    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:50.570331 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:50.569960    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:51.570738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:51.570702    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:51.571125 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:51.570766    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:52.570775 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:52.570740    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:52.571178 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:52.570819    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:53.570392 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:53.570353    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:53.570546 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:53.570406    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:54.570876 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:54.570834    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused" start-of-body=
Nov 03 18:56:54.571411 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:54.570884    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": dial tcp 10.134.0.31:15021: connect: connection refused"
Nov 03 18:56:54.848654 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:54.848544    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:55.098560 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:55.098519    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:56:56.300813 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:56.300763    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:56.301278 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:56.301254    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:56:56.301903 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:56:56.301883    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 18:57:00.199566 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:00.199531    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:00.444016 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:00.443986    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:03.724803 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.724764    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cf8d44fc4-w4tl2"]
Nov 03 18:57:03.735426 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.735396    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.780889 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.780859    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf8d44fc4-w4tl2"]
Nov 03 18:57:03.822219 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.822185    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkwj\" (UniqueName: \"kubernetes.io/projected/2c3a53ea-a998-4524-b09e-c0ef5faec585-kube-api-access-qdkwj\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.822362 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.822257    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3a53ea-a998-4524-b09e-c0ef5faec585-tls-cert\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.923428 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.923399    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3a53ea-a998-4524-b09e-c0ef5faec585-tls-cert\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.923615 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.923507    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkwj\" (UniqueName: \"kubernetes.io/projected/2c3a53ea-a998-4524-b09e-c0ef5faec585-kube-api-access-qdkwj\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.925811 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.925770    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2c3a53ea-a998-4524-b09e-c0ef5faec585-tls-cert\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.980894 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.980827    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkwj\" (UniqueName: \"kubernetes.io/projected/2c3a53ea-a998-4524-b09e-c0ef5faec585-kube-api-access-qdkwj\") pod \"authorino-6cf8d44fc4-w4tl2\" (UID: \"2c3a53ea-a998-4524-b09e-c0ef5faec585\") " pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:03.994415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:03.994389    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:04.047730 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:04.047699    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2"
Nov 03 18:57:04.266498 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:04.266468    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:04.412188 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:04.412153    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf8d44fc4-w4tl2"]
Nov 03 18:57:04.419955 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:57:04.419931    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c3a53ea_a998_4524_b09e_c0ef5faec585.slice/crio-aa445499ac86e02990606f3910e79ecf68df5661ecfe85054e763a654e75fed8 WatchSource:0}: Error finding container aa445499ac86e02990606f3910e79ecf68df5661ecfe85054e763a654e75fed8: Status 404 returned error can't find the container with id aa445499ac86e02990606f3910e79ecf68df5661ecfe85054e763a654e75fed8
Nov 03 18:57:04.711093 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:04.711012    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2" event={"ID":"2c3a53ea-a998-4524-b09e-c0ef5faec585","Type":"ContainerStarted","Data":"aa445499ac86e02990606f3910e79ecf68df5661ecfe85054e763a654e75fed8"}
Nov 03 18:57:05.223803 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:57:05.223762    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:57:06.722032 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:06.721991    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2" event={"ID":"2c3a53ea-a998-4524-b09e-c0ef5faec585","Type":"ContainerStarted","Data":"b0e7956e9226813a1addfb0d8ab8ee005eded36fe14aa5606a32fe1e001296f3"}
Nov 03 18:57:06.830666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:06.830600    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6cf8d44fc4-w4tl2" podStartSLOduration=2.455200687 podStartE2EDuration="3.830583033s" totalImagesPullingTime="1.375382346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:06.829453695 +0000 UTC m=+725.106807280" watchObservedRunningTime="2026-11-03 18:57:06.830583033 +0000 UTC m=+725.107936638"
Nov 03 18:57:06.878245 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:06.878207    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:57:06.878739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:06.878703    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" podUID="dcc69b60-b4e5-4427-a0c1-d8b352158343" containerName="authorino" containerID="cri-o://eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900" gracePeriod=30
Nov 03 18:57:07.134533 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.134502    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:57:07.253044 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.252953    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\" (UniqueName: \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\") pod \"dcc69b60-b4e5-4427-a0c1-d8b352158343\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") "
Nov 03 18:57:07.253274 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.253057    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\" (UniqueName: \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\") pod \"dcc69b60-b4e5-4427-a0c1-d8b352158343\" (UID: \"dcc69b60-b4e5-4427-a0c1-d8b352158343\") "
Nov 03 18:57:07.255129 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.255093    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv" pod "dcc69b60-b4e5-4427-a0c1-d8b352158343" (UID: "dcc69b60-b4e5-4427-a0c1-d8b352158343"). InnerVolumeSpecName "kube-api-access-gwbgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 18:57:07.276269 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.276232    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert" pod "dcc69b60-b4e5-4427-a0c1-d8b352158343" (UID: "dcc69b60-b4e5-4427-a0c1-d8b352158343"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 18:57:07.354491 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.354457    4944 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dcc69b60-b4e5-4427-a0c1-d8b352158343-tls-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:57:07.354491 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.354488    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbgv\" (UniqueName: \"kubernetes.io/projected/dcc69b60-b4e5-4427-a0c1-d8b352158343-kube-api-access-gwbgv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 18:57:07.732867 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.731879    4944 generic.go:356] "Generic (PLEG): container finished" podID="dcc69b60-b4e5-4427-a0c1-d8b352158343" containerID="eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900" exitCode=0
Nov 03 18:57:07.732867 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.732622    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v"
Nov 03 18:57:07.739968 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.739933    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" event={"ID":"dcc69b60-b4e5-4427-a0c1-d8b352158343","Type":"ContainerDied","Data":"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"}
Nov 03 18:57:07.740215 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.740197    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d69bd8d6f-k8f4v" event={"ID":"dcc69b60-b4e5-4427-a0c1-d8b352158343","Type":"ContainerDied","Data":"2445867cbf2f470b87d87f85b1f241198554916a2d2a9e45ecc3e59568d534a8"}
Nov 03 18:57:07.740382 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.740329    4944 scope.go:122] "RemoveContainer" containerID="eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"
Nov 03 18:57:07.752872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.752850    4944 scope.go:122] "RemoveContainer" containerID="eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"
Nov 03 18:57:07.753177 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:57:07.753158    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900\": container with ID starting with eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900 not found: ID does not exist" containerID="eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"
Nov 03 18:57:07.753245 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.753186    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900"} err="failed to get container status \"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900\": rpc error: code = NotFound desc = could not find container \"eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900\": container with ID starting with eb748781b5e73246f7a381f84cf3a37899b6c0121b8b708c42f9214dbb437900 not found: ID does not exist"
Nov 03 18:57:07.890970 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.890936    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:57:07.894666 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.894637    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d69bd8d6f-k8f4v"]
Nov 03 18:57:07.933187 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:07.933156    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc69b60-b4e5-4427-a0c1-d8b352158343" path="/var/lib/kubelet/pods/dcc69b60-b4e5-4427-a0c1-d8b352158343/volumes"
Nov 03 18:57:08.039024 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:08.038990    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:09.777997 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:09.777967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:10.021900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:10.021862    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:11.496258 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.496226    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 18:57:11.496618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.496596    4944 cpu_manager.go:414] "containerMap: removing container" podUID="dcc69b60-b4e5-4427-a0c1-d8b352158343" containerName="authorino"
Nov 03 18:57:11.496618 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.496609    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc69b60-b4e5-4427-a0c1-d8b352158343" containerName="authorino"
Nov 03 18:57:11.496692 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.496680    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc69b60-b4e5-4427-a0c1-d8b352158343" containerName="authorino"
Nov 03 18:57:11.500991 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.500974    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596513 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596482    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596513 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596516    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596758 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596539    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596758 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596559    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596758 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596620    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596941 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596823    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596941 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596864    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.596941 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596891    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpntq\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.597094 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.596973    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.597656 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.597634    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 18:57:11.698217 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698192    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698379 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698227    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698379 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698242    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpntq\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698379 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698369    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698524 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698485    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698582 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698522    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698582 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698565    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698690 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698589    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698690 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698613    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698690 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698616    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.698872 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.698826    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.699070 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.699046    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.699191 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.699133    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.699191 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.699143    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.700594 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.700574    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.701024 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.701007    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.755015 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.754955    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.755109 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.755064    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpntq\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\") pod \"maas-default-gateway-openshift-default-77f556d647-4pb7s\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:11.813315 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:11.813289    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:12.233955 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.233874    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 18:57:12.242183 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:57:12.242157    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e935542_6696_45c8_a529_21cabdd2b8ef.slice/crio-1b0876d895cb33a8e18f2626c7116708f7e64c0f25ba4a1263e8e4d4f0073749 WatchSource:0}: Error finding container 1b0876d895cb33a8e18f2626c7116708f7e64c0f25ba4a1263e8e4d4f0073749: Status 404 returned error can't find the container with id 1b0876d895cb33a8e18f2626c7116708f7e64c0f25ba4a1263e8e4d4f0073749
Nov 03 18:57:12.244069 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.244040    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:57:12.244140 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.244104    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:57:12.244140 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.244134    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:57:12.758171 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.758134    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" event={"ID":"5e935542-6696-45c8-a529-21cabdd2b8ef","Type":"ContainerStarted","Data":"ab17813fd02465dd682fab5ad5efc4f117b866edf111866b62c354dbb7aa0387"}
Nov 03 18:57:12.758672 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.758186    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" event={"ID":"5e935542-6696-45c8-a529-21cabdd2b8ef","Type":"ContainerStarted","Data":"1b0876d895cb33a8e18f2626c7116708f7e64c0f25ba4a1263e8e4d4f0073749"}
Nov 03 18:57:12.814966 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.814883    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:12.871638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:12.871585    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podStartSLOduration=1.8715691140000001 podStartE2EDuration="1.871569114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:12.870860463 +0000 UTC m=+731.148214047" watchObservedRunningTime="2026-11-03 18:57:12.871569114 +0000 UTC m=+731.148922697"
Nov 03 18:57:13.823599 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:13.823270    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:13.823599 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:13.823331    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:14.814188 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:14.814151    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:14.814468 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:14.814438    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:14.844314 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:14.844269    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:15.093391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.093275    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:15.295457 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.295427    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m756l"]
Nov 03 18:57:15.298623 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.298607    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.355521 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.355452    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.355735 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.355715    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j2gh\" (UniqueName: \"kubernetes.io/projected/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-kube-api-access-9j2gh\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.393127 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.393082    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 18:57:15.393127 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.393120    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8skxv\""
Nov 03 18:57:15.394612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.394587    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m756l"]
Nov 03 18:57:15.457079 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.457042    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.457250 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.457097    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j2gh\" (UniqueName: \"kubernetes.io/projected/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-kube-api-access-9j2gh\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.457750 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.457729    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.518810 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.517181    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j2gh\" (UniqueName: \"kubernetes.io/projected/aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a-kube-api-access-9j2gh\") pod \"payload-processing-5f78687d5b-m756l\" (UID: \"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a\") " pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.638103 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.638013    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:15.816620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.816167    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:15.816620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:15.816232    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:16.006136 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:16.006100    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m756l"]
Nov 03 18:57:16.029390 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:57:16.029360    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafa7ea1_c312_4a7b_8d9e_72cb169f0c6a.slice/crio-43e4f0b057ce3dbeff838a80808a1fde2cd951cd70fbedeb753273a130f8f7c8 WatchSource:0}: Error finding container 43e4f0b057ce3dbeff838a80808a1fde2cd951cd70fbedeb753273a130f8f7c8: Status 404 returned error can't find the container with id 43e4f0b057ce3dbeff838a80808a1fde2cd951cd70fbedeb753273a130f8f7c8
Nov 03 18:57:16.776357 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:16.776317    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-m756l" event={"ID":"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a","Type":"ContainerStarted","Data":"43e4f0b057ce3dbeff838a80808a1fde2cd951cd70fbedeb753273a130f8f7c8"}
Nov 03 18:57:16.814176 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:16.814140    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:16.814408 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:16.814382    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:17.815584 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:17.815142    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:17.815584 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:17.815206    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:18.814606 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:18.814566    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:18.814815 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:18.814635    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:19.586004 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.585971    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:19.794694 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.794654    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-m756l" event={"ID":"aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a","Type":"ContainerStarted","Data":"38c75c7c6f7c5fda70b90823700c8b401004745d5a071c765d48820ef3ed7d79"}
Nov 03 18:57:19.794896 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.794771    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:19.814649 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.814622    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 03 18:57:19.814719 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.814678    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 03 18:57:19.906822 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:19.906713    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-m756l" podStartSLOduration=1.844487137 podStartE2EDuration="4.906699402s" totalImagesPullingTime="3.062212265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:57:19.905227675 +0000 UTC m=+738.182581260" watchObservedRunningTime="2026-11-03 18:57:19.906699402 +0000 UTC m=+738.184052985"
Nov 03 18:57:21.854978 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:21.854936    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4pb7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 18:57:21.855402 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:21.855011    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 18:57:22.766426 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:22.766393    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:22.766936 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:22.766911    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:22.767761 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:22.767743    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 18:57:30.803425 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:30.803397    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-m756l"
Nov 03 18:57:35.223713 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:57:35.223678    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:57:37.766691 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:37.766661    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:38.250959 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:38.250924    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:39.803810 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:39.803759    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:40.054043 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:40.053956    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:41.324367 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:41.324296    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:41.576757 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:41.576683    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:43.815253 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:43.815220    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:57:44.064266 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:57:44.064216    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:05.223278 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:58:05.223241    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:58:08.166963 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:08.166915    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:08.426009 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:08.425927    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:08.822699 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:08.822666    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:08.939446 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:08.939413    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 18:58:08.946459 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:08.946439    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.006323 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.006292    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 18:58:09.032101 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032074    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032101 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032104    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032128    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032230    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032252    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v6w7\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032325    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032415 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032371    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.032514 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.032434    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.073096 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.073030    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:09.075679 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.075664    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-98860e-openshift-default-dockercfg-p56jx\""
Nov 03 18:58:09.133138 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133101    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133154    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133212    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133239    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133274    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133507 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133305    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133507 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133335    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v6w7\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133507 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133382    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133507 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133418    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133673 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133529    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133673 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133589    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.133905 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133885    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.134005 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.133948    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.134082 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.134057    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.135597 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.135581    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.135662 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.135646    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.190714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.190683    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.191195 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.190749    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v6w7\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.258644 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.258608    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:09.328829 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.328698    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:09.573888 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.573855    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 18:58:09.577534 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.577519    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.582478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.582419    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:09.626375 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.626352    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-619731-openshift-default-dockercfg-m7qhx\""
Nov 03 18:58:09.635120 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.635087    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 18:58:09.637738 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637709    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637868 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637754    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637868 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637820    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637868 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637855    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl88z\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637981 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637910    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637981 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637930    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.637981 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637948    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.638084 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.637982    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.638084 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.638002    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738449 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738419    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl88z\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738614 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738482    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738614 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738513    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738614 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738546    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738614 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738600    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738852 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738631    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738852 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738663    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738852 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738695    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.738852 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738750    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.739054 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738967    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.739054 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.738995    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.739171 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.739088    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.739223 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.739169    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.739465 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.739445    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.740885 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.740863    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.741125 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.741106    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.749798 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.749770    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 18:58:09.757569 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:09.757542    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2370d8f3_38ed_422a_8163_ea4684f20252.slice/crio-23d1746face52d753c6e4bf6706fa44251066d9bb1153996a793ef91a56c016f WatchSource:0}: Error finding container 23d1746face52d753c6e4bf6706fa44251066d9bb1153996a793ef91a56c016f: Status 404 returned error can't find the container with id 23d1746face52d753c6e4bf6706fa44251066d9bb1153996a793ef91a56c016f
Nov 03 18:58:09.759884 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.759849    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:09.759979 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.759915    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:09.759979 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.759943    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:09.802217 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.802192    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.802377 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.802191    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl88z\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.851922 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.851830    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:09.890729 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.890696    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:09.922229 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.922194    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 18:58:09.926739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.926717    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.940849 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.940827    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.940969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.940877    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.940969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.940899    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.940969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.940917    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.940969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.940961    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5n6\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.941443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.941411    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.941532 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.941491    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.941532 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.941523    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.941676 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.941554    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:09.978827 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.975480    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-010ed3-openshift-default-dockercfg-9tvsf\""
Nov 03 18:58:09.983824 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:09.983725    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 18:58:10.023578 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.023538    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerStarted","Data":"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"}
Nov 03 18:58:10.023759 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.023594    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerStarted","Data":"23d1746face52d753c6e4bf6706fa44251066d9bb1153996a793ef91a56c016f"}
Nov 03 18:58:10.042505 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042472    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5n6\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.042688 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042536    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.042688 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042656    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.042847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042734    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.042847 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042767    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.042962 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042915    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043019 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042958    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043019 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.042961    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043120 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043044    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043120 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043080    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043228 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043306 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043283    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043685 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043660    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.043866 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.043848    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.045300 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.045274    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.045385 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.045323    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.099782 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.099741    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.099940 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.099827    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5n6\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.109055 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.108974    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:10.152248 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.152194    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podStartSLOduration=2.152182278 podStartE2EDuration="2.152182278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:10.151000112 +0000 UTC m=+788.428353710" watchObservedRunningTime="2026-11-03 18:58:10.152182278 +0000 UTC m=+788.429535860"
Nov 03 18:58:10.243901 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.243862    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:10.259342 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.259318    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:10.264714 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.264692    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:10.269021 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.268997    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 18:58:10.277244 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:10.277188    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda822a620_0a77_49b0_b08f_ae0a750b90d9.slice/crio-4c11ea6ea19ec0e815d511048dac3128fcb944eac30e38e639b33c3ad6db8a22 WatchSource:0}: Error finding container 4c11ea6ea19ec0e815d511048dac3128fcb944eac30e38e639b33c3ad6db8a22: Status 404 returned error can't find the container with id 4c11ea6ea19ec0e815d511048dac3128fcb944eac30e38e639b33c3ad6db8a22
Nov 03 18:58:10.281308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.280343    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.281308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.280409    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.281308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.280448    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.361324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.361265    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:10.611639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.611555    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:10.622694 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.622661    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 18:58:10.631329 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:10.631297    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb540b40d_ecb8_4938_803c_38a23bf724fd.slice/crio-ea502e314a7b11e77d382b5a8820d1111ceb5ea890a11c27eedbde2b1f931743 WatchSource:0}: Error finding container ea502e314a7b11e77d382b5a8820d1111ceb5ea890a11c27eedbde2b1f931743: Status 404 returned error can't find the container with id ea502e314a7b11e77d382b5a8820d1111ceb5ea890a11c27eedbde2b1f931743
Nov 03 18:58:10.633828 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.633770    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.633912 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.633880    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.633951 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.633926    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:10.859853 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:10.859812    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:11.029319 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.029284    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerStarted","Data":"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"}
Nov 03 18:58:11.029519 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.029343    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerStarted","Data":"4c11ea6ea19ec0e815d511048dac3128fcb944eac30e38e639b33c3ad6db8a22"}
Nov 03 18:58:11.030874 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.030850    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerStarted","Data":"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"}
Nov 03 18:58:11.030978 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.030891    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerStarted","Data":"ea502e314a7b11e77d382b5a8820d1111ceb5ea890a11c27eedbde2b1f931743"}
Nov 03 18:58:11.031030 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.031015    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:11.032032 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.032016    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:11.106845 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.106818    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:11.144571 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.144521    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podStartSLOduration=2.144508793 podStartE2EDuration="2.144508793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:11.142552055 +0000 UTC m=+789.419905661" watchObservedRunningTime="2026-11-03 18:58:11.144508793 +0000 UTC m=+789.421862376"
Nov 03 18:58:11.245031 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.245001    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:11.249642 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.249619    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:11.256539 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.256493    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podStartSLOduration=2.256479931 podStartE2EDuration="2.256479931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:11.255568328 +0000 UTC m=+789.532921938" watchObservedRunningTime="2026-11-03 18:58:11.256479931 +0000 UTC m=+789.533833513"
Nov 03 18:58:11.890937 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.890910    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:11.896305 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:11.896280    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:12.035669 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:12.035641    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:12.035900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:12.035834    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:12.036531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:12.036510    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:12.036620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:12.036559    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:13.685593 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:13.685562    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:13.946392 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:13.946292    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:14.241080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:14.240979    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:14.490886 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:14.490857    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:14.905739 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:14.905706    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:15.162842 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:15.162741    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:15.435823 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:15.435726    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:15.689333 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:15.689264    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:15.939911 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:15.939839    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:16.199724 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:16.199645    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:18.193478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.193444    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:18.465995 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.465913    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:18.506391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.506363    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 18:58:18.509799 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.509770    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.604698 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.604669    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-98860e\""
Nov 03 18:58:18.604925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.604819    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-98860e-dockercfg-7f6pj\""
Nov 03 18:58:18.606543 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.606520    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 18:58:18.608066 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.608048    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 18:58:18.611674 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.611653    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.620522 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.620411    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.620621 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.620544    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4wp\" (UniqueName: \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.668465 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.668427    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 18:58:18.721861 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.721775    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4wp\" (UniqueName: \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.721988 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.721858    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.721988 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.721929    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqw46\" (UniqueName: \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.721988 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.721956    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.722557 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.722539    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.724669 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.724638    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:18.778901 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.778880    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4wp\" (UniqueName: \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\") pod \"payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.822498 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.822470    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqw46\" (UniqueName: \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.822639 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.822503    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.823155 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.823137    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.824147 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.824132    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:18.888821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.886435    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqw46\" (UniqueName: \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\") pod \"payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:18.928557 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:18.928525    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:19.196438 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:19.196406    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 18:58:19.203524 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:19.203498    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8592e418_c7a0_4c21_b5fb_719097dc10f5.slice/crio-3cb0dcff78883c19d4990e4b90fdc8c56521ca572debbf42e818f35c8fe12cca WatchSource:0}: Error finding container 3cb0dcff78883c19d4990e4b90fdc8c56521ca572debbf42e818f35c8fe12cca: Status 404 returned error can't find the container with id 3cb0dcff78883c19d4990e4b90fdc8c56521ca572debbf42e818f35c8fe12cca
Nov 03 18:58:19.295204 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:19.295177    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 18:58:19.301293 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:19.301254    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0f517d_2724_4285_b993_edfee82810d0.slice/crio-4d463a7e0079c8d38cc5b2c6bc5fbe2fed57678eb78fcdf76408cbd94ce4da6e WatchSource:0}: Error finding container 4d463a7e0079c8d38cc5b2c6bc5fbe2fed57678eb78fcdf76408cbd94ce4da6e: Status 404 returned error can't find the container with id 4d463a7e0079c8d38cc5b2c6bc5fbe2fed57678eb78fcdf76408cbd94ce4da6e
Nov 03 18:58:19.371006 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:19.370962    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:19.622613 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:19.622583    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:19.939150 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:19.939074    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:20.071174 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.071136    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" event={"ID":"7f0f517d-2724-4285-b993-edfee82810d0","Type":"ContainerStarted","Data":"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"}
Nov 03 18:58:20.071348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.071195    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" event={"ID":"7f0f517d-2724-4285-b993-edfee82810d0","Type":"ContainerStarted","Data":"4d463a7e0079c8d38cc5b2c6bc5fbe2fed57678eb78fcdf76408cbd94ce4da6e"}
Nov 03 18:58:20.071348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.071238    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:20.072740 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.072719    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" event={"ID":"8592e418-c7a0-4c21-b5fb-719097dc10f5","Type":"ContainerStarted","Data":"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"}
Nov 03 18:58:20.072851 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.072761    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" event={"ID":"8592e418-c7a0-4c21-b5fb-719097dc10f5","Type":"ContainerStarted","Data":"3cb0dcff78883c19d4990e4b90fdc8c56521ca572debbf42e818f35c8fe12cca"}
Nov 03 18:58:20.072888 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.072879    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:20.179970 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.179918    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" podStartSLOduration=2.179905357 podStartE2EDuration="2.179905357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:20.178807934 +0000 UTC m=+798.456161513" watchObservedRunningTime="2026-11-03 18:58:20.179905357 +0000 UTC m=+798.457258939"
Nov 03 18:58:20.289396 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:20.289349    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" podStartSLOduration=2.289337008 podStartE2EDuration="2.289337008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:20.287089315 +0000 UTC m=+798.564442898" watchObservedRunningTime="2026-11-03 18:58:20.289337008 +0000 UTC m=+798.566690590"
Nov 03 18:58:23.291089 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.289261    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:23.588674 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.588596    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:23.594836 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.594807    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 18:58:23.598376 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.598357    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.671299 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.671247    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k486z\" (UniqueName: \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.671630 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.671607    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.693265 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.693238    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-fa2878\""
Nov 03 18:58:23.693265 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.693238    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-fa2878-dockercfg-9kqhw\""
Nov 03 18:58:23.695237 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.695214    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 18:58:23.748296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.748266    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 18:58:23.751870 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.751850    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:23.772908 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.772888    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k486z\" (UniqueName: \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.772993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.772934    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.773531 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.773513    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.805734 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.805711    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 18:58:23.830099 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.830071    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k486z\" (UniqueName: \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\") pod \"payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.874333 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.874266    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:23.874443 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.874340    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lpmt\" (UniqueName: \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:23.911355 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.911327    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:23.976427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.976393    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:23.976556 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.976475    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lpmt\" (UniqueName: \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:23.977783 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:23.977755    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:24.034818 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.034774    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lpmt\" (UniqueName: \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\") pod \"payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:24.065807 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.065763    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:24.230888 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.230784    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:24.275885 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.275853    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 18:58:24.282975 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:24.282945    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaef37501_ac5d_4cfe_a36a_b9e8a3757148.slice/crio-a86318e24e84d3322205329c00f54ed1fe8dab01429622628e0ffff90db22911 WatchSource:0}: Error finding container a86318e24e84d3322205329c00f54ed1fe8dab01429622628e0ffff90db22911: Status 404 returned error can't find the container with id a86318e24e84d3322205329c00f54ed1fe8dab01429622628e0ffff90db22911
Nov 03 18:58:24.434977 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.434948    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 18:58:24.442348 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:24.442321    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b2f48_552b_4467_ae5c_09572c5200d4.slice/crio-9b7fe176d4bbee6ab1c17aca56499fc4eec8cdbbdb2458164a601eee994a4d25 WatchSource:0}: Error finding container 9b7fe176d4bbee6ab1c17aca56499fc4eec8cdbbdb2458164a601eee994a4d25: Status 404 returned error can't find the container with id 9b7fe176d4bbee6ab1c17aca56499fc4eec8cdbbdb2458164a601eee994a4d25
Nov 03 18:58:24.477448 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.477420    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:24.723751 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:24.723721    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:25.096227 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.096189    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" event={"ID":"aef37501-ac5d-4cfe-a36a-b9e8a3757148","Type":"ContainerStarted","Data":"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"}
Nov 03 18:58:25.096431 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.096243    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" event={"ID":"aef37501-ac5d-4cfe-a36a-b9e8a3757148","Type":"ContainerStarted","Data":"a86318e24e84d3322205329c00f54ed1fe8dab01429622628e0ffff90db22911"}
Nov 03 18:58:25.096431 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.096287    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:25.097904 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.097880    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" event={"ID":"f58b2f48-552b-4467-ae5c-09572c5200d4","Type":"ContainerStarted","Data":"f5385c30fb04f142b48e6da84db3f313087fec14273f062d6de97706dbe2f0c2"}
Nov 03 18:58:25.098012 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.097917    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" event={"ID":"f58b2f48-552b-4467-ae5c-09572c5200d4","Type":"ContainerStarted","Data":"9b7fe176d4bbee6ab1c17aca56499fc4eec8cdbbdb2458164a601eee994a4d25"}
Nov 03 18:58:25.098012 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.097940    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:25.204103 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.204048    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" podStartSLOduration=2.204030842 podStartE2EDuration="2.204030842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:25.203110207 +0000 UTC m=+803.480463802" watchObservedRunningTime="2026-11-03 18:58:25.204030842 +0000 UTC m=+803.481384425"
Nov 03 18:58:25.313644 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.313588    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" podStartSLOduration=2.3135737020000002 podStartE2EDuration="2.313573702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:25.311854976 +0000 UTC m=+803.589208562" watchObservedRunningTime="2026-11-03 18:58:25.313573702 +0000 UTC m=+803.590927284"
Nov 03 18:58:25.710954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.710923    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:25.973737 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:25.973653    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:26.231559 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.231478    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:26.231941 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.231922    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 18:58:26.235343 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.235312    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.288478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.288452    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-92596a\""
Nov 03 18:58:26.299809 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.299152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfnl4\" (UniqueName: \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.299809 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.299237    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.334737 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.334703    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-92596a-dockercfg-8zt8h\""
Nov 03 18:58:26.336587 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.336565    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 18:58:26.391478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.391447    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 18:58:26.395299 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.395279    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.400951 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.400922    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfnl4\" (UniqueName: \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.401092 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.400989    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.401869 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.401847    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.450236 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.450201    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 18:58:26.459161 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.459136    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfnl4\" (UniqueName: \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\") pod \"payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.501677 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.501653    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.501831 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.501695    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrxck\" (UniqueName: \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.548500 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.548474    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:26.602328 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.602289    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.602703 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.602357    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrxck\" (UniqueName: \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.603464 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.603435    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.659394 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.659364    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrxck\" (UniqueName: \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\") pod \"payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.708781 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.708748    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:26.916780 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:26.916745    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 18:58:26.923215 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:26.923181    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6295d3b4_4f91_47ad_bd49_0423dd596eb8.slice/crio-666cfe3e2e90b0e1b7f415229ea0827fd7ff9e3b61dcb0cce6b8ababb0d2e380 WatchSource:0}: Error finding container 666cfe3e2e90b0e1b7f415229ea0827fd7ff9e3b61dcb0cce6b8ababb0d2e380: Status 404 returned error can't find the container with id 666cfe3e2e90b0e1b7f415229ea0827fd7ff9e3b61dcb0cce6b8ababb0d2e380
Nov 03 18:58:27.077961 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.077869    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 18:58:27.084588 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:27.084562    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1a5694_d54b_4f71_ae61_6d57be897b37.slice/crio-14bdefcf85e0e164af929372d6c8d2e7081ffc7349c32425d62ae6366016b4e5 WatchSource:0}: Error finding container 14bdefcf85e0e164af929372d6c8d2e7081ffc7349c32425d62ae6366016b4e5: Status 404 returned error can't find the container with id 14bdefcf85e0e164af929372d6c8d2e7081ffc7349c32425d62ae6366016b4e5
Nov 03 18:58:27.109735 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.109702    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" event={"ID":"6295d3b4-4f91-47ad-bd49-0423dd596eb8","Type":"ContainerStarted","Data":"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"}
Nov 03 18:58:27.109870 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.109764    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" event={"ID":"6295d3b4-4f91-47ad-bd49-0423dd596eb8","Type":"ContainerStarted","Data":"666cfe3e2e90b0e1b7f415229ea0827fd7ff9e3b61dcb0cce6b8ababb0d2e380"}
Nov 03 18:58:27.109870 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.109822    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:27.110783 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.110761    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" event={"ID":"7a1a5694-d54b-4f71-ae61-6d57be897b37","Type":"ContainerStarted","Data":"14bdefcf85e0e164af929372d6c8d2e7081ffc7349c32425d62ae6366016b4e5"}
Nov 03 18:58:27.218612 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:27.218526    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" podStartSLOduration=1.218505761 podStartE2EDuration="1.218505761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:27.217587395 +0000 UTC m=+805.494941004" watchObservedRunningTime="2026-11-03 18:58:27.218505761 +0000 UTC m=+805.495859383"
Nov 03 18:58:28.136006 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.135970    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" event={"ID":"7a1a5694-d54b-4f71-ae61-6d57be897b37","Type":"ContainerStarted","Data":"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"}
Nov 03 18:58:28.136553 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.136390    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:28.243363 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.243129    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" podStartSLOduration=2.243110825 podStartE2EDuration="2.243110825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:28.242730418 +0000 UTC m=+806.520084012" watchObservedRunningTime="2026-11-03 18:58:28.243110825 +0000 UTC m=+806.520464408"
Nov 03 18:58:28.790478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.790446    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 18:58:28.794052 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.794037    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:28.842665 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.842637    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-619731\""
Nov 03 18:58:28.892119 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.892085    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-619731-dockercfg-kwndj\""
Nov 03 18:58:28.894099 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.894074    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 18:58:28.947439 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.947407    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:28.947620 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:28.947540    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srd96\" (UniqueName: \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.048551 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.048458    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srd96\" (UniqueName: \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.048701 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.048570    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.049271 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.049254    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.105688 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.105652    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srd96\" (UniqueName: \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\") pod \"payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.130825 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.129997    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:29.251629 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.251588    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:29.510817 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.508568    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 18:58:29.533701 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:29.533667    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbb7582_6d86_418a_8f25_c0af33550947.slice/crio-2b10c3eb525bcd753216a0b0407b3b95282f458e8ddf1873a7c036e6e68faad3 WatchSource:0}: Error finding container 2b10c3eb525bcd753216a0b0407b3b95282f458e8ddf1873a7c036e6e68faad3: Status 404 returned error can't find the container with id 2b10c3eb525bcd753216a0b0407b3b95282f458e8ddf1873a7c036e6e68faad3
Nov 03 18:58:29.635809 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.635549    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:29.937527 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:29.937448    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:30.148427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:30.148209    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" event={"ID":"adbb7582-6d86-418a-8f25-c0af33550947","Type":"ContainerStarted","Data":"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"}
Nov 03 18:58:30.148427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:30.148256    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" event={"ID":"adbb7582-6d86-418a-8f25-c0af33550947","Type":"ContainerStarted","Data":"2b10c3eb525bcd753216a0b0407b3b95282f458e8ddf1873a7c036e6e68faad3"}
Nov 03 18:58:30.148427 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:30.148324    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:30.255999 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:30.255955    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" podStartSLOduration=2.255940357 podStartE2EDuration="2.255940357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:30.255227092 +0000 UTC m=+808.532580676" watchObservedRunningTime="2026-11-03 18:58:30.255940357 +0000 UTC m=+808.533293941"
Nov 03 18:58:31.080229 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.080201    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 18:58:31.080380 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.080276    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 18:58:31.250716 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.250678    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:31.494845 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.494816    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:31.533710 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.533677    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 18:58:31.538308 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.538287    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.573958 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.573921    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 18:58:31.574073 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.573929    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.574073 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.574036    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkrd2\" (UniqueName: \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.586954 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.586915    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-010ed3\""
Nov 03 18:58:31.587174 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.587156    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-010ed3-dockercfg-fjw9p\""
Nov 03 18:58:31.669322 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.669291    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 18:58:31.672995 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.672976    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.675395 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.675373    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.675477 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.675417    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkrd2\" (UniqueName: \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.676124 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.676102    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.737687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.737656    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 18:58:31.738121 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.738098    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkrd2\" (UniqueName: \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\") pod \"payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.777224 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.777148    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgkj\" (UniqueName: \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.777362 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.777267    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.853976 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.853939    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:31.878303 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.878275    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.878442 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.878423    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgkj\" (UniqueName: \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.879286 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.879262    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.938332 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.938292    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgkj\" (UniqueName: \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\") pod \"payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:31.984993 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:31.984963    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:32.236337 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:32.235524    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 18:58:32.241243 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:32.241216    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4f7cde_10c0_4e15_86e4_de4cf5d6675e.slice/crio-e8891451b0cc51a6cea027424f3b31644e550f85e16da7029ce2a25fa0a44061 WatchSource:0}: Error finding container e8891451b0cc51a6cea027424f3b31644e550f85e16da7029ce2a25fa0a44061: Status 404 returned error can't find the container with id e8891451b0cc51a6cea027424f3b31644e550f85e16da7029ce2a25fa0a44061
Nov 03 18:58:32.378684 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:32.378608    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 18:58:32.385143 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:32.385115    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d0aa4d_e2ea_4a87_bcb6_11ad5826dafc.slice/crio-43b6d8bd3e8e8e2e5d3fac60c39ef3d4cf3573bd35432941da39e65fd0bd9b68 WatchSource:0}: Error finding container 43b6d8bd3e8e8e2e5d3fac60c39ef3d4cf3573bd35432941da39e65fd0bd9b68: Status 404 returned error can't find the container with id 43b6d8bd3e8e8e2e5d3fac60c39ef3d4cf3573bd35432941da39e65fd0bd9b68
Nov 03 18:58:33.162737 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.162700    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" event={"ID":"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc","Type":"ContainerStarted","Data":"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"}
Nov 03 18:58:33.163142 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.162759    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" event={"ID":"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc","Type":"ContainerStarted","Data":"43b6d8bd3e8e8e2e5d3fac60c39ef3d4cf3573bd35432941da39e65fd0bd9b68"}
Nov 03 18:58:33.163142 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.162809    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:33.168228 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.168193    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" event={"ID":"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e","Type":"ContainerStarted","Data":"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"}
Nov 03 18:58:33.168376 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.168251    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" event={"ID":"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e","Type":"ContainerStarted","Data":"e8891451b0cc51a6cea027424f3b31644e550f85e16da7029ce2a25fa0a44061"}
Nov 03 18:58:33.168376 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.168279    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:33.281377 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.281309    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" podStartSLOduration=2.281290786 podStartE2EDuration="2.281290786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:33.278946993 +0000 UTC m=+811.556300578" watchObservedRunningTime="2026-11-03 18:58:33.281290786 +0000 UTC m=+811.558644370"
Nov 03 18:58:33.390223 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:33.390160    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" podStartSLOduration=2.390139596 podStartE2EDuration="2.390139596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:33.386956234 +0000 UTC m=+811.664309816" watchObservedRunningTime="2026-11-03 18:58:33.390139596 +0000 UTC m=+811.667493180"
Nov 03 18:58:34.146333 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:34.146292    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:34.392911 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:34.392872    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:34.651004 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:34.650969    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:35.224136 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:58:35.224099    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:58:36.118067 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:36.116436    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 18:58:36.118067 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:36.116521    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 18:58:37.189753 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.189719    4944 generic.go:356] "Generic (PLEG): container finished" podID="2370d8f3-38ed-422a-8163-ea4684f20252" containerID="26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248" exitCode=0
Nov 03 18:58:37.190212 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.189802    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerDied","Data":"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"}
Nov 03 18:58:37.190212 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.190173    4944 scope.go:122] "RemoveContainer" containerID="26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"
Nov 03 18:58:37.191417 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.191389    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:37.191521 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.191471    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:37.191583 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:37.191517    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:38.160087 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.160049    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 18:58:38.197488 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.197456    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerStarted","Data":"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"}
Nov 03 18:58:38.263816 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.262485    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:38.263816 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.262696    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:38.263816 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.262823    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:38.576848 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.576771    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:38.834818 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:38.834645    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:39.049814 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.049396    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 18:58:39.061610 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.061586    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.075503 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.075166    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnrr\" (UniqueName: \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.075503 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.075259    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.076201 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.076163    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 18:58:39.130862 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.130117    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-eb6ef8\""
Nov 03 18:58:39.131023 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.130933    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-eb6ef8-dockercfg-khq2k\""
Nov 03 18:58:39.164521 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.164468    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 18:58:39.176529 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.176501    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnrr\" (UniqueName: \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.177080 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.177060    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.178085 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.178061    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.259593 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.259552    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:39.260186 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.260136    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:39.264815 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.263937    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnrr\" (UniqueName: \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\") pod \"payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.283461 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.283417    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:39.377261 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.377228    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:39.561471 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.561427    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:39.798410 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.798368    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 18:58:39.804528 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:39.804502    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d0db6f_bb1a_4fc0_aadc_6c6f88396b62.slice/crio-26a8c42fc38df937eb8c6de9e8935720faf6f0f48e9ae81a71cc29b208f65e65 WatchSource:0}: Error finding container 26a8c42fc38df937eb8c6de9e8935720faf6f0f48e9ae81a71cc29b208f65e65: Status 404 returned error can't find the container with id 26a8c42fc38df937eb8c6de9e8935720faf6f0f48e9ae81a71cc29b208f65e65
Nov 03 18:58:39.815050 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:39.814993    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:40.233887 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:40.215190    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" event={"ID":"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62","Type":"ContainerStarted","Data":"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"}
Nov 03 18:58:40.233887 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:40.215258    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" event={"ID":"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62","Type":"ContainerStarted","Data":"26a8c42fc38df937eb8c6de9e8935720faf6f0f48e9ae81a71cc29b208f65e65"}
Nov 03 18:58:40.233887 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:40.216125    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:40.259859 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:40.259823    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:40.260239 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:40.259870    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:41.038816 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:41.037916    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:41.157860 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:41.157812    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 18:58:41.259288 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:41.259246    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:41.259460 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:41.259312    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:41.267295 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:41.267244    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" podStartSLOduration=3.267230358 podStartE2EDuration="3.267230358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:40.33906078 +0000 UTC m=+818.616414363" watchObservedRunningTime="2026-11-03 18:58:41.267230358 +0000 UTC m=+819.544583941"
Nov 03 18:58:42.259296 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:42.259259    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:42.259589 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:42.259546    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:43.259005 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.258973    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 03 18:58:43.259359 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.259029    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 03 18:58:43.600090 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.600002    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:43.845922 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.845880    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:43.903862 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.903757    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 18:58:43.907389 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:43.907364    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.007279 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.007227    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-90a9bf\""
Nov 03 18:58:44.009977 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.009922    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-90a9bf-dockercfg-5nrgx\""
Nov 03 18:58:44.011235 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.011205    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 18:58:44.056358 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.056328    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 18:58:44.059977 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.059959    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.090686 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.090653    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.090883 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.090744    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvdc\" (UniqueName: \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.109663 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.109626    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 18:58:44.176508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.176428    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 18:58:44.176508 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.176499    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 18:58:44.191588 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.191555    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvdc\" (UniqueName: \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.191741 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.191619    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.191741 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.191699    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.191924 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.191900    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsd9g\" (UniqueName: \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.192414 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.192395    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.248814 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.248762    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvdc\" (UniqueName: \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\") pod \"payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.263725 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.263694    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:44.264984 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.264946    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 18:58:44.292641 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.292596    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.293045 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.293023    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsd9g\" (UniqueName: \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.293667 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.293638    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.353273 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.353231    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsd9g\" (UniqueName: \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\") pod \"payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.372912 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.372880    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:44.388849 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.388772    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:44.521821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.521357    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:44.757478 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.757444    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 18:58:44.764494 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:44.764469    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287445df_2f48_41d0_b80a_1149b7619398.slice/crio-c9fe23a6e96a413d7e19e85759d1bbfc4ef556b6f0e99fcbf722e419613f4478 WatchSource:0}: Error finding container c9fe23a6e96a413d7e19e85759d1bbfc4ef556b6f0e99fcbf722e419613f4478: Status 404 returned error can't find the container with id c9fe23a6e96a413d7e19e85759d1bbfc4ef556b6f0e99fcbf722e419613f4478
Nov 03 18:58:44.909450 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:44.909411    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 18:58:44.914887 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:58:44.914862    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc368dcb7_83e7_45ef_8202_6072c0250343.slice/crio-9f5d41409a1b8f8c137e7bd2af52747e63db355b2d5e4f94e607867c41d38be1 WatchSource:0}: Error finding container 9f5d41409a1b8f8c137e7bd2af52747e63db355b2d5e4f94e607867c41d38be1: Status 404 returned error can't find the container with id 9f5d41409a1b8f8c137e7bd2af52747e63db355b2d5e4f94e607867c41d38be1
Nov 03 18:58:45.280126 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.280088    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" event={"ID":"c368dcb7-83e7-45ef-8202-6072c0250343","Type":"ContainerStarted","Data":"73fef8062f85892a53f6af6dae0f6ef4fd682cd87f61d838623051721c649b97"}
Nov 03 18:58:45.280584 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.280148    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" event={"ID":"c368dcb7-83e7-45ef-8202-6072c0250343","Type":"ContainerStarted","Data":"9f5d41409a1b8f8c137e7bd2af52747e63db355b2d5e4f94e607867c41d38be1"}
Nov 03 18:58:45.280584 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.280256    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:45.281722 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.281702    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" event={"ID":"287445df-2f48-41d0-b80a-1149b7619398","Type":"ContainerStarted","Data":"3fba2c53ad484f1db241d621c6407e96927be5b32d936b44e43d1e62018a4237"}
Nov 03 18:58:45.281843 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.281734    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" event={"ID":"287445df-2f48-41d0-b80a-1149b7619398","Type":"ContainerStarted","Data":"c9fe23a6e96a413d7e19e85759d1bbfc4ef556b6f0e99fcbf722e419613f4478"}
Nov 03 18:58:45.281843 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.281822    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:45.388344 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.388278    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" podStartSLOduration=2.3882589100000002 podStartE2EDuration="2.38825891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:45.387223752 +0000 UTC m=+823.664577359" watchObservedRunningTime="2026-11-03 18:58:45.38825891 +0000 UTC m=+823.665612495"
Nov 03 18:58:45.496383 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:45.496323    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" podStartSLOduration=1.496309364 podStartE2EDuration="1.496309364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:58:45.494650763 +0000 UTC m=+823.772004347" watchObservedRunningTime="2026-11-03 18:58:45.496309364 +0000 UTC m=+823.773662947"
Nov 03 18:58:46.287985 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.287954    4944 generic.go:356] "Generic (PLEG): container finished" podID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerID="893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942" exitCode=0
Nov 03 18:58:46.288360 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.288029    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerDied","Data":"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"}
Nov 03 18:58:46.288555 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.288540    4944 scope.go:122] "RemoveContainer" containerID="893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"
Nov 03 18:58:46.290290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.290240    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:46.296162 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.290408    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:46.296162 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:46.290513    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:47.296391 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:47.296350    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerStarted","Data":"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"}
Nov 03 18:58:47.891562 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:47.891527    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:47.891746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:47.891628    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 18:58:47.891746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:47.891663    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 18:58:48.891607 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:48.891565    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 18:58:48.891990 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:48.891644    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 18:58:48.965180 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:48.965144    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:49.221840 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:49.221717    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:49.468218 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:49.468182    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:49.891101 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:49.891066    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 18:58:49.891280 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:49.891127    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 18:58:50.891392 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:50.891360    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 18:58:50.891783 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:50.891415    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 18:58:51.891324 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:51.891287    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 03 18:58:51.891503 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:51.891336    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 03 18:58:52.265370 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:52.265344    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 18:58:52.895729 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:52.895701    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:52.896206 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:52.896184    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:52.896762 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:52.896740    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 18:58:53.871537 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:53.871500    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:56.291118 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:56.291091    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 18:58:56.291477 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:56.291166    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 18:58:57.036820 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.036772    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:58:57.037032 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.036858    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:58:57.403438 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.403355    4944 generic.go:356] "Generic (PLEG): container finished" podID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerID="7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a" exitCode=0
Nov 03 18:58:57.403438 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.403426    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerDied","Data":"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"}
Nov 03 18:58:57.403899 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.403819    4944 scope.go:122] "RemoveContainer" containerID="7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"
Nov 03 18:58:57.405106 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.405077    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:57.405210 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.405140    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:57.405210 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.405179    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 18:58:57.595676 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:57.595641    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:58.349640 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:58.349608    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:58.432821 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:58.427687    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerStarted","Data":"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"}
Nov 03 18:58:58.605111 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:58.605014    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:59.040684 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.040649    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:59.244015 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.243979    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:58:59.244315 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.244293    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:58:59.244366 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.244343    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:58:59.656760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.656731    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:58:59.902765 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:58:59.902733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:00.244301 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:00.244263    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:59:00.244482 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:00.244322    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:59:01.244297 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:01.244258    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:59:01.244721 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:01.244317    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:59:02.244206 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:02.244177    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:59:02.244363 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:02.244221    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:59:03.244752 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:03.244716    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 03 18:59:03.245159 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:03.244772    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 03 18:59:03.872915 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:03.872878    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:04.117715 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:04.117685    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:04.248878 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:04.248847    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:59:04.249250 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:04.249134    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:59:04.249902 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:04.249886    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 18:59:05.223784 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:59:05.223747    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:59:08.896007 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:08.895966    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:09.364258 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:09.364222    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:09.975932 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:09.975879    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:10.231760 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:10.231680    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:14.103498 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:14.103419    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:15.498494 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:15.498461    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:16.120348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:16.120312    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:16.374705 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:16.374622    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:19.094260 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:19.094224    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:20.318969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:20.318934    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:20.559480 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:20.559450    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:20.808969 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:20.808939    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:21.237139 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:21.237063    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:21.487933 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:21.487852    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:24.269525 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:24.269491    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:25.856631 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:25.856589    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:26.455388 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:26.455353    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:26.702560 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:26.702529    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:29.240483 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:29.240441    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:29.481833 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:29.479302    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:33.432342 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:33.432312    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:33.746050 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:33.746021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:35.223643 ip-10-0-137-192 kubenswrapper[4944]: E1103 18:59:35.223601    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 18:59:37.198103 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:37.198070    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:38.526437 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:38.526394    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:39.278914 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:39.278876    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:39.878318 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:39.878288    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:40.130297 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:40.130190    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:43.133715 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:43.133681    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:46.622767 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.622731    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 18:59:46.625702 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.625687    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.720492 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.720462    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"kube-root-ca.crt\""
Nov 03 18:59:46.720492 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.720490    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"default-dockercfg-4b9g8\""
Nov 03 18:59:46.720687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.720508    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"simulator-w1-010ed3-kserve-self-signed-certs\""
Nov 03 18:59:46.720687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.720498    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"openshift-service-ca.crt\""
Nov 03 18:59:46.722697 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.722679    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 18:59:46.756109 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756085    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.756260 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756123    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.756260 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756151    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.756348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756276    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8cp\" (UniqueName: \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.756348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756323    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.756348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.756345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.816705 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.816653    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:46.857011 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.856977    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857177 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857028    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857177 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857125    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857180    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8cp\" (UniqueName: \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857226    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857290 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857250    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857455 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857379    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857455 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857437    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.857623 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.857605    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.859638 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.859617    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.859752 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.859736    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.915458 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.915373    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8cp\" (UniqueName: \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\") pod \"simulator-w1-010ed3-kserve-87467d74c-kn7rd\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") " pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:46.937123 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:46.937091    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:47.306815 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.306770    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 18:59:47.312707 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:59:47.312682    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb04e78ee_97ec_4c4c_8cbb_f2f5fd112f1e.slice/crio-0ecdb641e021b819241f03b452d0402a9b4e64c03870827d8fe6ecc66900d45e WatchSource:0}: Error finding container 0ecdb641e021b819241f03b452d0402a9b4e64c03870827d8fe6ecc66900d45e: Status 404 returned error can't find the container with id 0ecdb641e021b819241f03b452d0402a9b4e64c03870827d8fe6ecc66900d45e
Nov 03 18:59:47.330458 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.330432    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:47.691362 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.691271    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" event={"ID":"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e","Type":"ContainerStarted","Data":"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"}
Nov 03 18:59:47.691362 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.691321    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" event={"ID":"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e","Type":"ContainerStarted","Data":"0ecdb641e021b819241f03b452d0402a9b4e64c03870827d8fe6ecc66900d45e"}
Nov 03 18:59:47.800348 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.800299    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podStartSLOduration=1.800282758 podStartE2EDuration="1.800282758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:47.799778968 +0000 UTC m=+886.077132551" watchObservedRunningTime="2026-11-03 18:59:47.800282758 +0000 UTC m=+886.077636341"
Nov 03 18:59:47.932276 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.932238    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:47.937306 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:47.937282    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:48.180661 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:48.180627    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:52.478451 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:52.478414    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:56.093364 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.093328    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 18:59:56.096215 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.096194    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.144956 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.144929    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"kube-root-ca.crt\""
Nov 03 18:59:56.190851 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.190823    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"openshift-service-ca.crt\""
Nov 03 18:59:56.190851 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.190825    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"default-dockercfg-pzhxj\""
Nov 03 18:59:56.191033 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.190880    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"simulator-w6-eb6ef8-kserve-self-signed-certs\""
Nov 03 18:59:56.192941 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.192922    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 18:59:56.254540 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254509    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.254746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254573    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.254746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254662    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.254746 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254703    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.254900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254798    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4brdf\" (UniqueName: \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.254900 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.254877    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.295020 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.294984    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:56.355459 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355376    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355459 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355433    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355459 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355462    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355489    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355520    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4brdf\" (UniqueName: \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355687 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355584    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355845 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355824    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.355914 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355884    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.356000 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.355982    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.357776 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.357752    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.358153 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.358136    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.414576 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.414547    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4brdf\" (UniqueName: \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\") pod \"simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") " pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.707521 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.707440    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 18:59:56.937705 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.937674    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:56.950286 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:56.950263    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:57.077925 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.077875    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 18:59:57.123315 ip-10-0-137-192 kubenswrapper[4944]: W1103 18:59:57.123286    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99011f11_bdc6_42b1_b33c_dff9aa5ed8e8.slice/crio-09cf453243b011983599f827d34882c4c03b9ca5a659d0b1d4c4e140430b46c8 WatchSource:0}: Error finding container 09cf453243b011983599f827d34882c4c03b9ca5a659d0b1d4c4e140430b46c8: Status 404 returned error can't find the container with id 09cf453243b011983599f827d34882c4c03b9ca5a659d0b1d4c4e140430b46c8
Nov 03 18:59:57.313223 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.313188    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 18:59:57.743710 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.743679    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" event={"ID":"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8","Type":"ContainerStarted","Data":"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"}
Nov 03 18:59:57.743905 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.743721    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" event={"ID":"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8","Type":"ContainerStarted","Data":"09cf453243b011983599f827d34882c4c03b9ca5a659d0b1d4c4e140430b46c8"}
Nov 03 18:59:57.754882 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.754858    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 18:59:57.853316 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:57.853268    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podStartSLOduration=1.8532540979999998 podStartE2EDuration="1.853254098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:59:57.851941343 +0000 UTC m=+896.129294940" watchObservedRunningTime="2026-11-03 18:59:57.853254098 +0000 UTC m=+896.130607681"
Nov 03 18:59:58.708170 ip-10-0-137-192 kubenswrapper[4944]: I1103 18:59:58.708139    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:00:00.178604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.178556    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:00.181728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.181706    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:00.181866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.181839    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.184642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.184625    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:00.184762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.184743    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.187344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.187330    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.224113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.224087    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:00.226944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.226922    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:00.227052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.227034    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.229579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.229560    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.235374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.235352    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:00.243830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.243777    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:00.247233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.247210    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:00.268428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.268410    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:00:00.268538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.268461    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:00:00.281260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.281236    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:00.294771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.294742    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwcd\" (UniqueName: \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.294891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.294810    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.294936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.294919    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhmd\" (UniqueName: \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.295044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.295022    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxf87\" (UniqueName: \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.295081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.295067    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.295112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.295087    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.332936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.332911    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:00:00.342858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.342833    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:00.396430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396406    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxf87\" (UniqueName: \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.396570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396450    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.396643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396565    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.396643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396604    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fslmn\" (UniqueName: \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.396781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396647    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwcd\" (UniqueName: \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.396781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396664    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.396781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396692    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.396986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396854    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.396986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.396900    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqkn\" (UniqueName: \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.397114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.397024    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhmd\" (UniqueName: \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.397263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.397246    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.397328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.397269    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.397328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.397303    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.453929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.453865    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwcd\" (UniqueName: \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.454079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.454057    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhmd\" (UniqueName: \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\") pod \"maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.454184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.454163    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxf87\" (UniqueName: \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.496246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.496218    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:00.497627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.497607    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fslmn\" (UniqueName: \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.497693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.497650    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.497733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.497691    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.497733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.497715    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqkn\" (UniqueName: \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.498330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.498305    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.498421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.498378    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.503307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.503284    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:00.509095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.509073    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:00.556658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.556628    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqkn\" (UniqueName: \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.562350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.561438    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fslmn\" (UniqueName: \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.846744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.846709    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:00.854836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.854812    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:00.873610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.873580    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:00.882927 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:00.882898    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8d80cf_ee74_422c_abfc_49901216b07d.slice/crio-f1959fcba36795b53ed7c505f60a4a47048bdb3e52e40893234c29b154009a18 WatchSource:0}: Error finding container f1959fcba36795b53ed7c505f60a4a47048bdb3e52e40893234c29b154009a18: Status 404 returned error can't find the container with id f1959fcba36795b53ed7c505f60a4a47048bdb3e52e40893234c29b154009a18
Nov 03 19:00:00.928945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.928922    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:00.931865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:00.931807    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:00.933995 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:00.933966    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1f5f40_3372_44c7_b06f_eb8ec5caafd4.slice/crio-466113af512c02f85260abaa14901ac5f943af6d93f3958529ad59dfdc66f6c1 WatchSource:0}: Error finding container 466113af512c02f85260abaa14901ac5f943af6d93f3958529ad59dfdc66f6c1: Status 404 returned error can't find the container with id 466113af512c02f85260abaa14901ac5f943af6d93f3958529ad59dfdc66f6c1
Nov 03 19:00:00.936246 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:00.936054    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5e00c3_936c_460d_b0bc_eb81c2de4cb1.slice/crio-84bb12de4dfee7217fc6145e03fd093d5fce9728ebb8241cca74719f52a04580 WatchSource:0}: Error finding container 84bb12de4dfee7217fc6145e03fd093d5fce9728ebb8241cca74719f52a04580: Status 404 returned error can't find the container with id 84bb12de4dfee7217fc6145e03fd093d5fce9728ebb8241cca74719f52a04580
Nov 03 19:00:01.217087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.217007    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:01.222966 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:01.222923    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88986f28_2c6b_4385_8bb9_e2bcf10ed4aa.slice/crio-8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515 WatchSource:0}: Error finding container 8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515: Status 404 returned error can't find the container with id 8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515
Nov 03 19:00:01.224784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.224764    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:01.230577 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:01.230552    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c044afc_4220_4962_ae60_f85e6dd0c020.slice/crio-43192df9a5c8139f9a9943eecd152d9b80e7bed9fb59e76aa4cbb306d4ed6a8f WatchSource:0}: Error finding container 43192df9a5c8139f9a9943eecd152d9b80e7bed9fb59e76aa4cbb306d4ed6a8f: Status 404 returned error can't find the container with id 43192df9a5c8139f9a9943eecd152d9b80e7bed9fb59e76aa4cbb306d4ed6a8f
Nov 03 19:00:01.566215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.566179    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:01.767061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.767027    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" event={"ID":"5c044afc-4220-4962-ae60-f85e6dd0c020","Type":"ContainerStarted","Data":"43192df9a5c8139f9a9943eecd152d9b80e7bed9fb59e76aa4cbb306d4ed6a8f"}
Nov 03 19:00:01.768194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.768164    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerStarted","Data":"466113af512c02f85260abaa14901ac5f943af6d93f3958529ad59dfdc66f6c1"}
Nov 03 19:00:01.769297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.769277    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" event={"ID":"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa","Type":"ContainerStarted","Data":"8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515"}
Nov 03 19:00:01.770402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.770381    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" event={"ID":"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1","Type":"ContainerStarted","Data":"84bb12de4dfee7217fc6145e03fd093d5fce9728ebb8241cca74719f52a04580"}
Nov 03 19:00:01.771480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:01.771461    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerStarted","Data":"f1959fcba36795b53ed7c505f60a4a47048bdb3e52e40893234c29b154009a18"}
Nov 03 19:00:02.020045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:02.020019    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:00:02.021715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:02.021689    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:00:05.235493 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:05.235446    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:00:05.766527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:05.766496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:06.159246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.159164    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:06.707904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.707773    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:00:06.720589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.720564    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:00:06.804345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.804314    4944 generic.go:356] "Generic (PLEG): container finished" podID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerID="33f4b439f0ce4dbd876928251e03f1040b36f1e1733849cf485c44db96db7ffe" exitCode=60
Nov 03 19:00:06.804519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.804388    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" event={"ID":"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa","Type":"ContainerDied","Data":"33f4b439f0ce4dbd876928251e03f1040b36f1e1733849cf485c44db96db7ffe"}
Nov 03 19:00:06.804650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.804634    4944 scope.go:122] "RemoveContainer" containerID="33f4b439f0ce4dbd876928251e03f1040b36f1e1733849cf485c44db96db7ffe"
Nov 03 19:00:06.806030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.806006    4944 generic.go:356] "Generic (PLEG): container finished" podID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerID="db1ead86b2adea2222840048abffea43220b0a1f054b249f14e9673b62c2c783" exitCode=60
Nov 03 19:00:06.806149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.806063    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" event={"ID":"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1","Type":"ContainerDied","Data":"db1ead86b2adea2222840048abffea43220b0a1f054b249f14e9673b62c2c783"}
Nov 03 19:00:06.806292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.806274    4944 scope.go:122] "RemoveContainer" containerID="db1ead86b2adea2222840048abffea43220b0a1f054b249f14e9673b62c2c783"
Nov 03 19:00:06.807879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.807858    4944 generic.go:356] "Generic (PLEG): container finished" podID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerID="fd520a57c4bd508e7dffc883c7ca0311a5d524790e643fbeb88fc01efddf35d5" exitCode=60
Nov 03 19:00:06.807977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.807890    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" event={"ID":"5c044afc-4220-4962-ae60-f85e6dd0c020","Type":"ContainerDied","Data":"fd520a57c4bd508e7dffc883c7ca0311a5d524790e643fbeb88fc01efddf35d5"}
Nov 03 19:00:06.808117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.808100    4944 scope.go:122] "RemoveContainer" containerID="fd520a57c4bd508e7dffc883c7ca0311a5d524790e643fbeb88fc01efddf35d5"
Nov 03 19:00:06.810670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.809719    4944 generic.go:356] "Generic (PLEG): container finished" podID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerID="9dcd215ade695e623fcff48d3573b09ebae713d11ddfe47f69ea6ea6a889bc12" exitCode=60
Nov 03 19:00:06.810670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.809826    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerDied","Data":"9dcd215ade695e623fcff48d3573b09ebae713d11ddfe47f69ea6ea6a889bc12"}
Nov 03 19:00:06.810670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.809985    4944 scope.go:122] "RemoveContainer" containerID="9dcd215ade695e623fcff48d3573b09ebae713d11ddfe47f69ea6ea6a889bc12"
Nov 03 19:00:06.824175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:06.824155    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:00:07.818774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.818742    4944 generic.go:356] "Generic (PLEG): container finished" podID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerID="f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1" exitCode=60
Nov 03 19:00:07.819204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.818886    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" event={"ID":"5c044afc-4220-4962-ae60-f85e6dd0c020","Type":"ContainerDied","Data":"f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1"}
Nov 03 19:00:07.819204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.818944    4944 scope.go:122] "RemoveContainer" containerID="fd520a57c4bd508e7dffc883c7ca0311a5d524790e643fbeb88fc01efddf35d5"
Nov 03 19:00:07.819204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.819190    4944 scope.go:122] "RemoveContainer" containerID="f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1"
Nov 03 19:00:07.819526 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:07.819498    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc_odh-ai-gateway-infra(5c044afc-4220-4962-ae60-f85e6dd0c020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020"
Nov 03 19:00:07.821155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.821131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerStarted","Data":"b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"}
Nov 03 19:00:07.823217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.823192    4944 generic.go:356] "Generic (PLEG): container finished" podID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerID="3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292" exitCode=60
Nov 03 19:00:07.823317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.823274    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" event={"ID":"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa","Type":"ContainerDied","Data":"3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292"}
Nov 03 19:00:07.823509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.823488    4944 scope.go:122] "RemoveContainer" containerID="3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292"
Nov 03 19:00:07.823721 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:07.823700    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q_odh-ai-gateway-infra(88986f28-2c6b-4385-8bb9-e2bcf10ed4aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa"
Nov 03 19:00:07.825293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.825273    4944 generic.go:356] "Generic (PLEG): container finished" podID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerID="c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40" exitCode=60
Nov 03 19:00:07.825385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.825355    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" event={"ID":"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1","Type":"ContainerDied","Data":"c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40"}
Nov 03 19:00:07.825597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.825578    4944 scope.go:122] "RemoveContainer" containerID="c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40"
Nov 03 19:00:07.825857 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:07.825839    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt_odh-ai-gateway-infra(9e5e00c3-936c-460d-b0bc-eb81c2de4cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1"
Nov 03 19:00:07.890691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.890671    4944 scope.go:122] "RemoveContainer" containerID="33f4b439f0ce4dbd876928251e03f1040b36f1e1733849cf485c44db96db7ffe"
Nov 03 19:00:07.902996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:07.902975    4944 scope.go:122] "RemoveContainer" containerID="db1ead86b2adea2222840048abffea43220b0a1f054b249f14e9673b62c2c783"
Nov 03 19:00:08.249408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.249355    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" podStartSLOduration=3.818420392 podStartE2EDuration="8.249340395s" totalImagesPullingTime="4.430920003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:08.247556419 +0000 UTC m=+906.524910002" watchObservedRunningTime="2026-11-03 19:00:08.249340395 +0000 UTC m=+906.526694006"
Nov 03 19:00:08.832571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.832540    4944 scope.go:122] "RemoveContainer" containerID="f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1"
Nov 03 19:00:08.833057 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:08.832836    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc_odh-ai-gateway-infra(5c044afc-4220-4962-ae60-f85e6dd0c020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020"
Nov 03 19:00:08.834277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.834255    4944 generic.go:356] "Generic (PLEG): container finished" podID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerID="b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c" exitCode=60
Nov 03 19:00:08.834359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.834336    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerDied","Data":"b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"}
Nov 03 19:00:08.834409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.834395    4944 scope.go:122] "RemoveContainer" containerID="9dcd215ade695e623fcff48d3573b09ebae713d11ddfe47f69ea6ea6a889bc12"
Nov 03 19:00:08.834519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.834505    4944 scope.go:122] "RemoveContainer" containerID="b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"
Nov 03 19:00:08.834745 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:08.834722    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk_odh-ai-gateway-infra(ee1f5f40-3372-44c7-b06f-eb8ec5caafd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4"
Nov 03 19:00:08.836121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.836098    4944 scope.go:122] "RemoveContainer" containerID="3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292"
Nov 03 19:00:08.836341 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:08.836319    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q_odh-ai-gateway-infra(88986f28-2c6b-4385-8bb9-e2bcf10ed4aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa"
Nov 03 19:00:08.837624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:08.837609    4944 scope.go:122] "RemoveContainer" containerID="c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40"
Nov 03 19:00:08.837827 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:08.837780    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt_odh-ai-gateway-infra(9e5e00c3-936c-460d-b0bc-eb81c2de4cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1"
Nov 03 19:00:09.844322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:09.844298    4944 scope.go:122] "RemoveContainer" containerID="b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"
Nov 03 19:00:09.844674 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:09.844544    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk_odh-ai-gateway-infra(ee1f5f40-3372-44c7-b06f-eb8ec5caafd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4"
Nov 03 19:00:10.690723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:10.690690    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:12.865025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:12.864989    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerStarted","Data":"b9addd25208dc58d22501a1f8f4eea9e00cd2e20759494da451500bce04a1633"}
Nov 03 19:00:12.976405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:12.976352    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" podStartSLOduration=1.509568724 podStartE2EDuration="12.976337649s" totalImagesPullingTime="11.466768925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:12.975314426 +0000 UTC m=+911.252668009" watchObservedRunningTime="2026-11-03 19:00:12.976337649 +0000 UTC m=+911.253691231"
Nov 03 19:00:13.871206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:13.871171    4944 generic.go:356] "Generic (PLEG): container finished" podID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerID="b9addd25208dc58d22501a1f8f4eea9e00cd2e20759494da451500bce04a1633" exitCode=60
Nov 03 19:00:13.871735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:13.871257    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerDied","Data":"b9addd25208dc58d22501a1f8f4eea9e00cd2e20759494da451500bce04a1633"}
Nov 03 19:00:13.871735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:13.871474    4944 scope.go:122] "RemoveContainer" containerID="b9addd25208dc58d22501a1f8f4eea9e00cd2e20759494da451500bce04a1633"
Nov 03 19:00:14.877961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:14.877932    4944 generic.go:356] "Generic (PLEG): container finished" podID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerID="092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d" exitCode=60
Nov 03 19:00:14.878333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:14.877971    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerDied","Data":"092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d"}
Nov 03 19:00:14.878333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:14.878031    4944 scope.go:122] "RemoveContainer" containerID="b9addd25208dc58d22501a1f8f4eea9e00cd2e20759494da451500bce04a1633"
Nov 03 19:00:14.878333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:14.878274    4944 scope.go:122] "RemoveContainer" containerID="092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d"
Nov 03 19:00:14.878554 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:14.878529    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw_odh-ai-gateway-infra(1b8d80cf-ee74-422c-abfc-49901216b07d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d"
Nov 03 19:00:15.357313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:15.357285    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:15.597999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:15.597948    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:15.885901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:15.885868    4944 scope.go:122] "RemoveContainer" containerID="092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d"
Nov 03 19:00:15.886289 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:15.886111    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw_odh-ai-gateway-infra(1b8d80cf-ee74-422c-abfc-49901216b07d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d"
Nov 03 19:00:17.927602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:17.927576    4944 scope.go:122] "RemoveContainer" containerID="b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"
Nov 03 19:00:18.903565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.903481    4944 generic.go:356] "Generic (PLEG): container finished" podID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerID="1631ca00897ad1d2b90675985a3ba9a2a3559ab7ccbf5f70d46b905b6b59765e" exitCode=60
Nov 03 19:00:18.903702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.903555    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerDied","Data":"1631ca00897ad1d2b90675985a3ba9a2a3559ab7ccbf5f70d46b905b6b59765e"}
Nov 03 19:00:18.903702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.903612    4944 scope.go:122] "RemoveContainer" containerID="b1793806186b58551a5c77e0171e7ca270c853996eb72ffed15b1654ebb65e6c"
Nov 03 19:00:18.903973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.903958    4944 scope.go:122] "RemoveContainer" containerID="1631ca00897ad1d2b90675985a3ba9a2a3559ab7ccbf5f70d46b905b6b59765e"
Nov 03 19:00:18.904226 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:18.904204    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk_odh-ai-gateway-infra(ee1f5f40-3372-44c7-b06f-eb8ec5caafd4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4"
Nov 03 19:00:18.927534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.927505    4944 scope.go:122] "RemoveContainer" containerID="f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1"
Nov 03 19:00:18.927674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:18.927658    4944 scope.go:122] "RemoveContainer" containerID="3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292"
Nov 03 19:00:19.915163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.915129    4944 generic.go:356] "Generic (PLEG): container finished" podID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerID="e9248292db04f8156e13943269c4344245d8547e945af0494d11dd2fc48a460d" exitCode=60
Nov 03 19:00:19.915360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.915205    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" event={"ID":"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa","Type":"ContainerDied","Data":"e9248292db04f8156e13943269c4344245d8547e945af0494d11dd2fc48a460d"}
Nov 03 19:00:19.915360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.915279    4944 scope.go:122] "RemoveContainer" containerID="3ea44bc3503b41e49f4ee41bd9d72529797858878f07670a2b1ddf71407e0292"
Nov 03 19:00:19.915644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.915627    4944 scope.go:122] "RemoveContainer" containerID="e9248292db04f8156e13943269c4344245d8547e945af0494d11dd2fc48a460d"
Nov 03 19:00:19.915966 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:19.915932    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q_odh-ai-gateway-infra(88986f28-2c6b-4385-8bb9-e2bcf10ed4aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa"
Nov 03 19:00:19.917161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.917138    4944 generic.go:356] "Generic (PLEG): container finished" podID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerID="bfd560e80aa07722a08b7458be5c8675a71ac2ae9ad11e65c17f2c14b988ea20" exitCode=60
Nov 03 19:00:19.917245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.917211    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" event={"ID":"5c044afc-4220-4962-ae60-f85e6dd0c020","Type":"ContainerDied","Data":"bfd560e80aa07722a08b7458be5c8675a71ac2ae9ad11e65c17f2c14b988ea20"}
Nov 03 19:00:19.917572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.917545    4944 scope.go:122] "RemoveContainer" containerID="bfd560e80aa07722a08b7458be5c8675a71ac2ae9ad11e65c17f2c14b988ea20"
Nov 03 19:00:19.917846 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:19.917828    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc_odh-ai-gateway-infra(5c044afc-4220-4962-ae60-f85e6dd0c020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020"
Nov 03 19:00:19.927270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.927241    4944 scope.go:122] "RemoveContainer" containerID="c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40"
Nov 03 19:00:19.940315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:19.940289    4944 scope.go:122] "RemoveContainer" containerID="f0a30d517b58f3bf114ca32046a3a4841170c5059b17f1a39068309fa758f8a1"
Nov 03 19:00:20.002622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.002590    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:20.023712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.023669    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:20.232730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.232709    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:20.298301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.298271    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\" (UniqueName: \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\") pod \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") "
Nov 03 19:00:20.298458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.298391    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\") pod \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\" (UID: \"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4\") "
Nov 03 19:00:20.298737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.298714    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca" pod "ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" (UID: "ee1f5f40-3372-44c7-b06f-eb8ec5caafd4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:20.300482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.300462    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87" pod "ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" (UID: "ee1f5f40-3372-44c7-b06f-eb8ec5caafd4"). InnerVolumeSpecName "kube-api-access-fxf87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:20.399648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.399617    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:20.399648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.399644    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxf87\" (UniqueName: \"kubernetes.io/projected/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4-kube-api-access-fxf87\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:20.693098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.693065    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:20.927015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.926982    4944 generic.go:356] "Generic (PLEG): container finished" podID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerID="8427a8f6f77e732de4e079ed6092e0a15ff6246f154bcd837ec21fa20d35fdb7" exitCode=60
Nov 03 19:00:20.927192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.927054    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" event={"ID":"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1","Type":"ContainerDied","Data":"8427a8f6f77e732de4e079ed6092e0a15ff6246f154bcd837ec21fa20d35fdb7"}
Nov 03 19:00:20.927192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.927109    4944 scope.go:122] "RemoveContainer" containerID="c1488433a16965673ad0ebdfea31b0f327037ecdae5e2b495888be7d3cf3ab40"
Nov 03 19:00:20.927484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.927461    4944 scope.go:122] "RemoveContainer" containerID="8427a8f6f77e732de4e079ed6092e0a15ff6246f154bcd837ec21fa20d35fdb7"
Nov 03 19:00:20.927761 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:20.927736    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt_odh-ai-gateway-infra(9e5e00c3-936c-460d-b0bc-eb81c2de4cb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1"
Nov 03 19:00:20.930026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.930004    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk" event={"ID":"ee1f5f40-3372-44c7-b06f-eb8ec5caafd4","Type":"ContainerDied","Data":"466113af512c02f85260abaa14901ac5f943af6d93f3958529ad59dfdc66f6c1"}
Nov 03 19:00:20.930124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.930011    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"
Nov 03 19:00:20.939101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:20.938894    4944 scope.go:122] "RemoveContainer" containerID="1631ca00897ad1d2b90675985a3ba9a2a3559ab7ccbf5f70d46b905b6b59765e"
Nov 03 19:00:21.035297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.035265    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:21.139445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.139411    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:21.143419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.143392    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895540-n48xk"]
Nov 03 19:00:21.165122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.165096    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:21.209928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.209865    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\" (UniqueName: \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\") pod \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") "
Nov 03 19:00:21.210045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.210037    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\") pod \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\" (UID: \"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa\") "
Nov 03 19:00:21.210383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.210352    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca" pod "88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" (UID: "88986f28-2c6b-4385-8bb9-e2bcf10ed4aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:21.211879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.211860    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn" pod "88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" (UID: "88986f28-2c6b-4385-8bb9-e2bcf10ed4aa"). InnerVolumeSpecName "kube-api-access-ctqkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:21.250484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.250457    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:21.312485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.312454    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqkn\" (UniqueName: \"kubernetes.io/projected/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-kube-api-access-ctqkn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:21.312485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.312485    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:21.395800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.395760    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:21.514596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.514572    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\" (UniqueName: \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\") pod \"5c044afc-4220-4962-ae60-f85e6dd0c020\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") "
Nov 03 19:00:21.514760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.514630    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\") pod \"5c044afc-4220-4962-ae60-f85e6dd0c020\" (UID: \"5c044afc-4220-4962-ae60-f85e6dd0c020\") "
Nov 03 19:00:21.515046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.515022    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca" pod "5c044afc-4220-4962-ae60-f85e6dd0c020" (UID: "5c044afc-4220-4962-ae60-f85e6dd0c020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:21.516665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.516649    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn" pod "5c044afc-4220-4962-ae60-f85e6dd0c020" (UID: "5c044afc-4220-4962-ae60-f85e6dd0c020"). InnerVolumeSpecName "kube-api-access-fslmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:21.616351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.616321    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fslmn\" (UniqueName: \"kubernetes.io/projected/5c044afc-4220-4962-ae60-f85e6dd0c020-kube-api-access-fslmn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:21.616351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.616348    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c044afc-4220-4962-ae60-f85e6dd0c020-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:21.937351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.937276    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" path="/var/lib/kubelet/pods/ee1f5f40-3372-44c7-b06f-eb8ec5caafd4/volumes"
Nov 03 19:00:21.939614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.939585    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q" event={"ID":"88986f28-2c6b-4385-8bb9-e2bcf10ed4aa","Type":"ContainerDied","Data":"8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515"}
Nov 03 19:00:21.939841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.939641    4944 scope.go:122] "RemoveContainer" containerID="e9248292db04f8156e13943269c4344245d8547e945af0494d11dd2fc48a460d"
Nov 03 19:00:21.939841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.939595    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"
Nov 03 19:00:21.946813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.943931    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc" event={"ID":"5c044afc-4220-4962-ae60-f85e6dd0c020","Type":"ContainerDied","Data":"43192df9a5c8139f9a9943eecd152d9b80e7bed9fb59e76aa4cbb306d4ed6a8f"}
Nov 03 19:00:21.946813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.943953    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"
Nov 03 19:00:21.959144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:21.959127    4944 scope.go:122] "RemoveContainer" containerID="bfd560e80aa07722a08b7458be5c8675a71ac2ae9ad11e65c17f2c14b988ea20"
Nov 03 19:00:21.988504 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:21.988477    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88986f28_2c6b_4385_8bb9_e2bcf10ed4aa.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c044afc_4220_4962_ae60_f85e6dd0c020.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88986f28_2c6b_4385_8bb9_e2bcf10ed4aa.slice/crio-8a1be1911f5abb99997b37251c19dbc9ba6bb82494a6ab011b26cab54daed515\": RecentStats: unable to find data in memory cache]"
Nov 03 19:00:22.047186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.047147    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:22.092254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.092220    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:22.096850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.096817    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895540-c7lvc"]
Nov 03 19:00:22.191817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.191753    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:22.199464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.199417    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:22.202124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.202104    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895540-nln6q"]
Nov 03 19:00:22.324608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.324575    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\" (UniqueName: \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\") pod \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") "
Nov 03 19:00:22.324762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.324636    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\") pod \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\" (UID: \"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1\") "
Nov 03 19:00:22.325004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.324980    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca" pod "9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" (UID: "9e5e00c3-936c-460d-b0bc-eb81c2de4cb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:22.325180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.325158    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:22.326642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.326615    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd" pod "9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" (UID: "9e5e00c3-936c-460d-b0bc-eb81c2de4cb1"). InnerVolumeSpecName "kube-api-access-wjwcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:22.426110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.426079    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwcd\" (UniqueName: \"kubernetes.io/projected/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1-kube-api-access-wjwcd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:22.958956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.958923    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"
Nov 03 19:00:22.958956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.958949    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt" event={"ID":"9e5e00c3-936c-460d-b0bc-eb81c2de4cb1","Type":"ContainerDied","Data":"84bb12de4dfee7217fc6145e03fd093d5fce9728ebb8241cca74719f52a04580"}
Nov 03 19:00:22.959227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:22.959005    4944 scope.go:122] "RemoveContainer" containerID="8427a8f6f77e732de4e079ed6092e0a15ff6246f154bcd837ec21fa20d35fdb7"
Nov 03 19:00:23.111471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:23.111437    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:23.157634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:23.157599    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895540-jx2jt"]
Nov 03 19:00:23.933855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:23.933821    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" path="/var/lib/kubelet/pods/5c044afc-4220-4962-ae60-f85e6dd0c020/volumes"
Nov 03 19:00:23.934243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:23.934230    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" path="/var/lib/kubelet/pods/88986f28-2c6b-4385-8bb9-e2bcf10ed4aa/volumes"
Nov 03 19:00:23.934574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:23.934563    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" path="/var/lib/kubelet/pods/9e5e00c3-936c-460d-b0bc-eb81c2de4cb1/volumes"
Nov 03 19:00:24.480380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:24.480349    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:26.928063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:26.928030    4944 scope.go:122] "RemoveContainer" containerID="092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d"
Nov 03 19:00:27.987665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:27.987581    4944 generic.go:356] "Generic (PLEG): container finished" podID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerID="b0af547cbe9708a2fb1a491ed801734f8b8d086c8976f609f5cafffa9c2c7110" exitCode=60
Nov 03 19:00:27.988105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:27.987659    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerDied","Data":"b0af547cbe9708a2fb1a491ed801734f8b8d086c8976f609f5cafffa9c2c7110"}
Nov 03 19:00:27.988105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:27.987724    4944 scope.go:122] "RemoveContainer" containerID="092a66894334b27707f0c5a51668d8f89ec737d1b4633a8f9e7e35fe1827ab3d"
Nov 03 19:00:27.988105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:27.988100    4944 scope.go:122] "RemoveContainer" containerID="b0af547cbe9708a2fb1a491ed801734f8b8d086c8976f609f5cafffa9c2c7110"
Nov 03 19:00:27.988396 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:27.988374    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw_odh-ai-gateway-infra(1b8d80cf-ee74-422c-abfc-49901216b07d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d"
Nov 03 19:00:28.971145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:28.971107    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:29.107070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.107034    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:29.250881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.250858    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:29.408284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.408254    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\" (UniqueName: \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\") pod \"1b8d80cf-ee74-422c-abfc-49901216b07d\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") "
Nov 03 19:00:29.408453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.408355    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\") pod \"1b8d80cf-ee74-422c-abfc-49901216b07d\" (UID: \"1b8d80cf-ee74-422c-abfc-49901216b07d\") "
Nov 03 19:00:29.408715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.408691    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca" pod "1b8d80cf-ee74-422c-abfc-49901216b07d" (UID: "1b8d80cf-ee74-422c-abfc-49901216b07d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:00:29.408864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.408846    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d80cf-ee74-422c-abfc-49901216b07d-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:29.410402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.410379    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd" pod "1b8d80cf-ee74-422c-abfc-49901216b07d" (UID: "1b8d80cf-ee74-422c-abfc-49901216b07d"). InnerVolumeSpecName "kube-api-access-wdhmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:00:29.509966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:29.509937    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhmd\" (UniqueName: \"kubernetes.io/projected/1b8d80cf-ee74-422c-abfc-49901216b07d-kube-api-access-wdhmd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:00:30.000244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.000217    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"
Nov 03 19:00:30.000244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.000225    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw" event={"ID":"1b8d80cf-ee74-422c-abfc-49901216b07d","Type":"ContainerDied","Data":"f1959fcba36795b53ed7c505f60a4a47048bdb3e52e40893234c29b154009a18"}
Nov 03 19:00:30.000424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.000278    4944 scope.go:122] "RemoveContainer" containerID="b0af547cbe9708a2fb1a491ed801734f8b8d086c8976f609f5cafffa9c2c7110"
Nov 03 19:00:30.153802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.153756    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:30.199176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.199101    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-eb6ef8-29895540-nxlgw"]
Nov 03 19:00:30.561723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.561694    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:00:30.562202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562188    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562203    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562216    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562221    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562230    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562234    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562240    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562244    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562259    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562263    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562269    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562273    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562285    4944 cpu_manager.go:414] "containerMap: removing container" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562290    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562299    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562304    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562309    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562314    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562321    4944 cpu_manager.go:414] "containerMap: removing container" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562325    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562331    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562335    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562343    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562347    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562353    4944 cpu_manager.go:414] "containerMap: removing container" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562357    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562425    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562432    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562436    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562443    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562447    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562453    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562459    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562464    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562469    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562474    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5e00c3-936c-460d-b0bc-eb81c2de4cb1" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562482    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:00:30.562510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.562487    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:00:30.565874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.565847    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.661450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.661403    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"kube-root-ca.crt\""
Nov 03 19:00:30.661450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.661430    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"default-dockercfg-pt8rk\""
Nov 03 19:00:30.661664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.661490    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"simulator-w0-92596a-kserve-self-signed-certs\""
Nov 03 19:00:30.661664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.661498    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"openshift-service-ca.crt\""
Nov 03 19:00:30.663511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.663494    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724392    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhfl5\" (UniqueName: \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724516    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724654    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724754    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724863    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.725077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.724890    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.773934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.773906    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:30.825812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.825718    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.825812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.825757    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.825860    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhfl5\" (UniqueName: \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.825935    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.825961    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.826035    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.826168    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.826246    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.826401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.826373    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.828135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.828114    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.828453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.828433    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:30.883396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:30.883363    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhfl5\" (UniqueName: \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\") pod \"simulator-w0-92596a-kserve-6856c59459-szqtz\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") " pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:31.176983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:31.176904    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:31.544627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:31.544596    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:00:31.550194 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:00:31.550166    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6e3ffa1_cb61_46e7_9131_82f3bd6273dc.slice/crio-4427b4fb2dc3cfb23b8da7a4853700b020455b262a2aaac2a5c8daa99035232c WatchSource:0}: Error finding container 4427b4fb2dc3cfb23b8da7a4853700b020455b262a2aaac2a5c8daa99035232c: Status 404 returned error can't find the container with id 4427b4fb2dc3cfb23b8da7a4853700b020455b262a2aaac2a5c8daa99035232c
Nov 03 19:00:31.935048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:31.934971    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8d80cf-ee74-422c-abfc-49901216b07d" path="/var/lib/kubelet/pods/1b8d80cf-ee74-422c-abfc-49901216b07d/volumes"
Nov 03 19:00:32.013407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:32.013373    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" event={"ID":"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc","Type":"ContainerStarted","Data":"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"}
Nov 03 19:00:32.013407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:32.013415    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" event={"ID":"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc","Type":"ContainerStarted","Data":"4427b4fb2dc3cfb23b8da7a4853700b020455b262a2aaac2a5c8daa99035232c"}
Nov 03 19:00:32.143066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:32.143007    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podStartSLOduration=2.142993279 podStartE2EDuration="2.142993279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:00:32.141706226 +0000 UTC m=+930.419059857" watchObservedRunningTime="2026-11-03 19:00:32.142993279 +0000 UTC m=+930.420346861"
Nov 03 19:00:32.177321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:32.177292    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:33.489072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:33.489039    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:35.224260 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:00:35.224228    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:00:38.146295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:38.146260    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:41.177438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:41.177349    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:41.190023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:41.189997    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:42.079558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:42.079532    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:00:42.710037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:42.710000    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:47.264892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:47.264858    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:47.621699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:47.621605    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:51.792176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:51.792141    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:00:56.276559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:00:56.276523    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:00.797752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:00.797702    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:01.404036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404001    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:01:01.404533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404518    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:01:01.404533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404535    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:01:01.404633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404554    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:01:01.404633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404560    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:01:01.404699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404641    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c044afc-4220-4962-ae60-f85e6dd0c020" containerName="cleanup"
Nov 03 19:01:01.404699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404651    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="88986f28-2c6b-4385-8bb9-e2bcf10ed4aa" containerName="cleanup"
Nov 03 19:01:01.404699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.404663    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1f5f40-3372-44c7-b06f-eb8ec5caafd4" containerName="cleanup"
Nov 03 19:01:01.408460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.408437    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.459250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.459202    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"premium-w2-619731-kserve-self-signed-certs\""
Nov 03 19:01:01.505397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.505362    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"default-dockercfg-tk8tf\""
Nov 03 19:01:01.505579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.505462    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"openshift-service-ca.crt\""
Nov 03 19:01:01.505579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.505525    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"kube-root-ca.crt\""
Nov 03 19:01:01.507800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.507771    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:01:01.546732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.546701    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.546912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.546759    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.546912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.546872    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.547029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.546919    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.547086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.547042    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lp78\" (UniqueName: \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.547139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.547080    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.602844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.602811    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:01.648018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.647985    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648034    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648077    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lp78\" (UniqueName: \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648096    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648157    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648218    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648478    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648511    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.648600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.648580    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.650455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.650432    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.650722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.650703    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.705687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.705612    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lp78\" (UniqueName: \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\") pod \"premium-w2-619731-kserve-87dfc47f5-lss4l\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") " pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:01.722511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:01.722476    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:02.097335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:02.097303    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:01:02.103077 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:02.103050    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15e1f75f_cf6e_44f6_b08c_c2eb20760e00.slice/crio-6341a0c370061ba63e932bca164194230281e2c1d5a56653d4b0e653bd7a3429 WatchSource:0}: Error finding container 6341a0c370061ba63e932bca164194230281e2c1d5a56653d4b0e653bd7a3429: Status 404 returned error can't find the container with id 6341a0c370061ba63e932bca164194230281e2c1d5a56653d4b0e653bd7a3429
Nov 03 19:01:02.177294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:02.177256    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" event={"ID":"15e1f75f-cf6e-44f6-b08c-c2eb20760e00","Type":"ContainerStarted","Data":"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"}
Nov 03 19:01:02.177419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:02.177314    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" event={"ID":"15e1f75f-cf6e-44f6-b08c-c2eb20760e00","Type":"ContainerStarted","Data":"6341a0c370061ba63e932bca164194230281e2c1d5a56653d4b0e653bd7a3429"}
Nov 03 19:01:02.289251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:02.289197    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podStartSLOduration=1.289184235 podStartE2EDuration="1.289184235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:02.287040126 +0000 UTC m=+960.564393736" watchObservedRunningTime="2026-11-03 19:01:02.289184235 +0000 UTC m=+960.566537817"
Nov 03 19:01:02.723142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:02.723113    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:05.223668 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:01:05.223628    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:01:05.444238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:05.444206    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:07.791957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.791920    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"]
Nov 03 19:01:07.796146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.796122    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:07.893057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.893025    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:01:07.893057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.893033    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:01:07.893300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.893284    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:01:07.893351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.893290    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:01:07.895711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.895684    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"]
Nov 03 19:01:07.912736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.912703    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:07.912907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.912756    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:07.912975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:07.912955    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscgg\" (UniqueName: \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.014271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.014228    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscgg\" (UniqueName: \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.014476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.014328    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.014476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.014351    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.015013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.014979    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.016703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.016682    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.071688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.071613    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscgg\" (UniqueName: \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\") pod \"maas-api-revoke-sub-858316dc-wghj6\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.109870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.109821    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:08.484183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:08.484106    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"]
Nov 03 19:01:08.490610 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:08.490575    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aebb74d_54ce_4830_9271_1861c20a863a.slice/crio-ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b WatchSource:0}: Error finding container ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b: Status 404 returned error can't find the container with id ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b
Nov 03 19:01:09.215585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:09.215500    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6" event={"ID":"7aebb74d-54ce-4830-9271-1861c20a863a","Type":"ContainerStarted","Data":"b7fb00dce7c523fdb9f1fcc0950c66e0de816bdab9bf2e9c7efbfe8f941f4215"}
Nov 03 19:01:09.215585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:09.215563    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6" event={"ID":"7aebb74d-54ce-4830-9271-1861c20a863a","Type":"ContainerStarted","Data":"ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b"}
Nov 03 19:01:09.329532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:09.325171    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6" podStartSLOduration=2.325151022 podStartE2EDuration="2.325151022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:09.32439349 +0000 UTC m=+967.601747075" watchObservedRunningTime="2026-11-03 19:01:09.325151022 +0000 UTC m=+967.602504607"
Nov 03 19:01:09.965629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:09.965593    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:10.222819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.222719    4944 generic.go:356] "Generic (PLEG): container finished" podID="7aebb74d-54ce-4830-9271-1861c20a863a" containerID="b7fb00dce7c523fdb9f1fcc0950c66e0de816bdab9bf2e9c7efbfe8f941f4215" exitCode=0
Nov 03 19:01:10.223235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.222812    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6" event={"ID":"7aebb74d-54ce-4830-9271-1861c20a863a","Type":"ContainerDied","Data":"b7fb00dce7c523fdb9f1fcc0950c66e0de816bdab9bf2e9c7efbfe8f941f4215"}
Nov 03 19:01:10.488912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.488825    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:01:10.492806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.492773    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.545703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.545538    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"premium-w1-010ed3-kserve-self-signed-certs\""
Nov 03 19:01:10.547490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.547462    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:01:10.656144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656113    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.656352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdxm\" (UniqueName: \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.656352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656259    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.656352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656311    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.656484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656383    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.656484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.656437    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.686409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.686381    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:10.757538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757514    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757602    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757623    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdxm\" (UniqueName: \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757649    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757672    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757693    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.757951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.757932    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.758052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.758032    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.758107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.758098    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.759928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.759907    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.760071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.760052    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:10.814593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:10.814557    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdxm\" (UniqueName: \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\") pod \"premium-w1-010ed3-kserve-564c67c8b6-k8gwz\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") " pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:11.104977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.104889    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:11.355080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.355017    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:11.464370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.464336    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\") pod \"7aebb74d-54ce-4830-9271-1861c20a863a\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") "
Nov 03 19:01:11.464570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.464427    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\") pod \"7aebb74d-54ce-4830-9271-1861c20a863a\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") "
Nov 03 19:01:11.464570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.464518    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\" (UniqueName: \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\") pod \"7aebb74d-54ce-4830-9271-1861c20a863a\" (UID: \"7aebb74d-54ce-4830-9271-1861c20a863a\") "
Nov 03 19:01:11.464968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.464877    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca" pod "7aebb74d-54ce-4830-9271-1861c20a863a" (UID: "7aebb74d-54ce-4830-9271-1861c20a863a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:01:11.465231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.465192    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:11.467948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.467903    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca" pod "7aebb74d-54ce-4830-9271-1861c20a863a" (UID: "7aebb74d-54ce-4830-9271-1861c20a863a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:01:11.468568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.468544    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg" pod "7aebb74d-54ce-4830-9271-1861c20a863a" (UID: "7aebb74d-54ce-4830-9271-1861c20a863a"). InnerVolumeSpecName "kube-api-access-qscgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:01:11.511570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.511532    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:01:11.521972 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:11.521941    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8290513e_d8eb_4c44_87cc_2e566616437b.slice/crio-ea5222d5f53581584635158bdf901744cad633382ebadbb545c10f944c79c852 WatchSource:0}: Error finding container ea5222d5f53581584635158bdf901744cad633382ebadbb545c10f944c79c852: Status 404 returned error can't find the container with id ea5222d5f53581584635158bdf901744cad633382ebadbb545c10f944c79c852
Nov 03 19:01:11.566552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.566529    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qscgg\" (UniqueName: \"kubernetes.io/projected/7aebb74d-54ce-4830-9271-1861c20a863a-kube-api-access-qscgg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:11.566552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.566553    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aebb74d-54ce-4830-9271-1861c20a863a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:11.723635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.723559    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:11.736078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:11.736051    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:12.235422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.235395    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"
Nov 03 19:01:12.235592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.235394    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6" event={"ID":"7aebb74d-54ce-4830-9271-1861c20a863a","Type":"ContainerDied","Data":"ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b"}
Nov 03 19:01:12.235592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.235519    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee2dcae1e52aee8dbf2de159af77301e5fd2762aab54a60c88e337f8d24c795b"
Nov 03 19:01:12.237049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.237013    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" event={"ID":"8290513e-d8eb-4c44-87cc-2e566616437b","Type":"ContainerStarted","Data":"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"}
Nov 03 19:01:12.237049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.237048    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" event={"ID":"8290513e-d8eb-4c44-87cc-2e566616437b","Type":"ContainerStarted","Data":"ea5222d5f53581584635158bdf901744cad633382ebadbb545c10f944c79c852"}
Nov 03 19:01:12.248748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.248720    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:01:12.346015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:12.345968    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podStartSLOduration=2.34595485 podStartE2EDuration="2.34595485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:12.34551691 +0000 UTC m=+970.622870606" watchObservedRunningTime="2026-11-03 19:01:12.34595485 +0000 UTC m=+970.623308434"
Nov 03 19:01:13.105113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:13.105079    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:14.478159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:14.478099    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:19.077752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:19.077694    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:21.037372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:21.037342    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:21.105528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:21.105498    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:21.118348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:21.118295    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:21.298586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:21.298501    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:01:23.543483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:23.543449    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:24.527503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:24.527464    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:25.176559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.176518    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:01:25.177115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.177095    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7aebb74d-54ce-4830-9271-1861c20a863a" containerName="revoke-keys"
Nov 03 19:01:25.177115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.177114    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aebb74d-54ce-4830-9271-1861c20a863a" containerName="revoke-keys"
Nov 03 19:01:25.177245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.177199    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aebb74d-54ce-4830-9271-1861c20a863a" containerName="revoke-keys"
Nov 03 19:01:25.180553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.180529    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.217373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.217332    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:25.230037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.230007    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fa2878-openshift-default-dockercfg-hfvcf\""
Nov 03 19:01:25.281319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.281282    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:01:25.314823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.314758    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr95k\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.314899    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.314968    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315062    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315088    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315125    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315141    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315160    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.315348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.315208    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416400    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416456    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416500    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416566    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416624    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr95k\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416682    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416709    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416859    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.416924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.416894    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.417833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.417806    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.418984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.418958    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.420262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.420201    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.421723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.421697    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.421841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.421815    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.423301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.423277    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.423896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.423874    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.475777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.475695    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.476032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.476006    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr95k\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.495457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.495426    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:25.938925 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:25.938897    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dad976e_f8e9_438a_898a_e8e4d0df20be.slice/crio-a6caf01da9a15b8924382571e18c601b526948c56d2f2964b9376c5be22f7f97 WatchSource:0}: Error finding container a6caf01da9a15b8924382571e18c601b526948c56d2f2964b9376c5be22f7f97: Status 404 returned error can't find the container with id a6caf01da9a15b8924382571e18c601b526948c56d2f2964b9376c5be22f7f97
Nov 03 19:01:25.939425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.939397    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:01:25.941813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.941743    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:25.941921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.941858    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:25.941921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:25.941904    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:26.323831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:26.323770    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" event={"ID":"8dad976e-f8e9-438a-898a-e8e4d0df20be","Type":"ContainerStarted","Data":"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"}
Nov 03 19:01:26.324205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:26.323879    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" event={"ID":"8dad976e-f8e9-438a-898a-e8e4d0df20be","Type":"ContainerStarted","Data":"a6caf01da9a15b8924382571e18c601b526948c56d2f2964b9376c5be22f7f97"}
Nov 03 19:01:26.496206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:26.496168    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:27.498089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:27.498051    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:27.498454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:27.498102    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:28.358781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:28.358745    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:28.496129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:28.496091    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:28.496295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:28.496158    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:29.496709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:29.496671    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:29.497228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:29.496744    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:30.496286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:30.496250    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:30.496465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:30.496323    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:31.496460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:31.496426    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:31.496863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:31.496490    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:32.496365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:32.496315    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 19:01:32.496572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:32.496379    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 19:01:32.995395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:32.995357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:33.500051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:33.500021    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:33.500484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:33.500304    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:33.501125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:33.501108    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:01:33.661341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:33.661281    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podStartSLOduration=8.661267248 podStartE2EDuration="8.661267248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:26.448284433 +0000 UTC m=+984.725638019" watchObservedRunningTime="2026-11-03 19:01:33.661267248 +0000 UTC m=+991.938620901"
Nov 03 19:01:35.224387 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:01:35.224338    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:01:37.503377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:37.503345    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:37.940901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:37.940821    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"]
Nov 03 19:01:37.946349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:37.946324    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.041582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.041542    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:01:38.041756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.041590    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:01:38.041756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.041590    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:01:38.041756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.041657    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:01:38.044034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.044009    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"]
Nov 03 19:01:38.052350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.052325    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.052782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.052752    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkd6\" (UniqueName: \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.052932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.052858    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.154341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.154302    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkd6\" (UniqueName: \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.154341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.154340    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.154584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.154375    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.155209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.155180    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.157227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.157197    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.211492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.211422    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkd6\" (UniqueName: \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\") pod \"maas-api-revoke-sub-3860d506-q774f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.259944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.259912    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:38.637259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:38.637225    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"]
Nov 03 19:01:38.643261 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:38.643238    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab33c8a7_d830_4e2c_a4ae_470e4a78ae2f.slice/crio-c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db WatchSource:0}: Error finding container c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db: Status 404 returned error can't find the container with id c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db
Nov 03 19:01:39.408310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:39.408272    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f" event={"ID":"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f","Type":"ContainerStarted","Data":"76d77c6965ea2b8a3db0303934e145419846445c82fd62374a2d663ae03db7f9"}
Nov 03 19:01:39.408506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:39.408359    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f" event={"ID":"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f","Type":"ContainerStarted","Data":"c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db"}
Nov 03 19:01:39.518346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:39.518292    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f" podStartSLOduration=2.518278574 podStartE2EDuration="2.518278574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:39.516203329 +0000 UTC m=+997.793556911" watchObservedRunningTime="2026-11-03 19:01:39.518278574 +0000 UTC m=+997.795632157"
Nov 03 19:01:40.415248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:40.415216    4944 generic.go:356] "Generic (PLEG): container finished" podID="ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" containerID="76d77c6965ea2b8a3db0303934e145419846445c82fd62374a2d663ae03db7f9" exitCode=0
Nov 03 19:01:40.415688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:40.415307    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f" event={"ID":"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f","Type":"ContainerDied","Data":"76d77c6965ea2b8a3db0303934e145419846445c82fd62374a2d663ae03db7f9"}
Nov 03 19:01:41.570138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.570113    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:41.692346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.692261    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\") pod \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") "
Nov 03 19:01:41.692502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.692361    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\") pod \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") "
Nov 03 19:01:41.692502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.692468    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\" (UniqueName: \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\") pod \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\" (UID: \"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f\") "
Nov 03 19:01:41.692751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.692723    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca" pod "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" (UID: "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:01:41.694418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.694391    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca" pod "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" (UID: "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:01:41.694569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.694547    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6" pod "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" (UID: "ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f"). InnerVolumeSpecName "kube-api-access-xzkd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:01:41.794225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.794185    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:41.794225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.794220    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkd6\" (UniqueName: \"kubernetes.io/projected/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-kube-api-access-xzkd6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:41.794225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:41.794234    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:42.000923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:42.000873    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:42.435506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:42.435471    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"
Nov 03 19:01:42.435506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:42.435480    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f" event={"ID":"ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f","Type":"ContainerDied","Data":"c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db"}
Nov 03 19:01:42.435718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:42.435565    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8741b1cdd6696982e5b16acc81f888ed7b69bf61eaf7ede00d390b9675123db"
Nov 03 19:01:46.594084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:46.594053    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:50.377568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:50.377535    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:51.086343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:51.086314    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:52.938498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:52.938459    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"]
Nov 03 19:01:52.938958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:52.938945    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" containerName="revoke-keys"
Nov 03 19:01:52.939004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:52.938958    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" containerName="revoke-keys"
Nov 03 19:01:52.939073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:52.939063    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" containerName="revoke-keys"
Nov 03 19:01:52.942279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:52.942263    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.040884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.040772    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:01:53.040884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.040826    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:01:53.040884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.040858    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:01:53.040884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.040891    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:01:53.043413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.043389    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"]
Nov 03 19:01:53.114408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.114376    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x64p\" (UniqueName: \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.114583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.114451    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.114583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.114541    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.215386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.215312    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x64p\" (UniqueName: \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.215526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.215418    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.215526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.215472    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.216088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.216064    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.217803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.217763    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.272777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.272743    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x64p\" (UniqueName: \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\") pod \"maas-api-revoke-sub-83b2cdfa-2wsqr\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.554864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.554831    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:53.923963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:53.923882    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"]
Nov 03 19:01:53.930117 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:53.930088    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb48a47d9_4c6d_4b40_85d6_8c769c952843.slice/crio-0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9 WatchSource:0}: Error finding container 0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9: Status 404 returned error can't find the container with id 0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9
Nov 03 19:01:54.502679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:54.502635    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr" event={"ID":"b48a47d9-4c6d-4b40-85d6-8c769c952843","Type":"ContainerStarted","Data":"0f89bee52a7f72353ed5fe89b63eeafcdcb5e5f8d61fc579f7d4dedf3980ba9f"}
Nov 03 19:01:54.503095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:54.502727    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr" event={"ID":"b48a47d9-4c6d-4b40-85d6-8c769c952843","Type":"ContainerStarted","Data":"0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9"}
Nov 03 19:01:54.705757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:54.705702    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr" podStartSLOduration=2.705690984 podStartE2EDuration="2.705690984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:54.705506474 +0000 UTC m=+1012.982860057" watchObservedRunningTime="2026-11-03 19:01:54.705690984 +0000 UTC m=+1012.983044567"
Nov 03 19:01:55.512544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:55.512512    4944 generic.go:356] "Generic (PLEG): container finished" podID="b48a47d9-4c6d-4b40-85d6-8c769c952843" containerID="0f89bee52a7f72353ed5fe89b63eeafcdcb5e5f8d61fc579f7d4dedf3980ba9f" exitCode=0
Nov 03 19:01:55.512961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:55.512550    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr" event={"ID":"b48a47d9-4c6d-4b40-85d6-8c769c952843","Type":"ContainerDied","Data":"0f89bee52a7f72353ed5fe89b63eeafcdcb5e5f8d61fc579f7d4dedf3980ba9f"}
Nov 03 19:01:55.857145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:55.857038    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:01:56.665444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.665419    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:56.753295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.753263    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\") pod \"b48a47d9-4c6d-4b40-85d6-8c769c952843\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") "
Nov 03 19:01:56.753452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.753403    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\" (UniqueName: \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\") pod \"b48a47d9-4c6d-4b40-85d6-8c769c952843\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") "
Nov 03 19:01:56.753520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.753463    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\") pod \"b48a47d9-4c6d-4b40-85d6-8c769c952843\" (UID: \"b48a47d9-4c6d-4b40-85d6-8c769c952843\") "
Nov 03 19:01:56.753888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.753857    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca" pod "b48a47d9-4c6d-4b40-85d6-8c769c952843" (UID: "b48a47d9-4c6d-4b40-85d6-8c769c952843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:01:56.755611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.755584    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca" pod "b48a47d9-4c6d-4b40-85d6-8c769c952843" (UID: "b48a47d9-4c6d-4b40-85d6-8c769c952843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:01:56.755713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.755608    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p" pod "b48a47d9-4c6d-4b40-85d6-8c769c952843" (UID: "b48a47d9-4c6d-4b40-85d6-8c769c952843"). InnerVolumeSpecName "kube-api-access-7x64p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:01:56.854567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.854493    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x64p\" (UniqueName: \"kubernetes.io/projected/b48a47d9-4c6d-4b40-85d6-8c769c952843-kube-api-access-7x64p\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:56.854567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.854521    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:56.854567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:56.854532    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b48a47d9-4c6d-4b40-85d6-8c769c952843-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:01:57.068418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.068383    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:01:57.068985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.068966    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b48a47d9-4c6d-4b40-85d6-8c769c952843" containerName="revoke-keys"
Nov 03 19:01:57.069058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.068988    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48a47d9-4c6d-4b40-85d6-8c769c952843" containerName="revoke-keys"
Nov 03 19:01:57.069094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.069076    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48a47d9-4c6d-4b40-85d6-8c769c952843" containerName="revoke-keys"
Nov 03 19:01:57.072760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.072740    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.156924    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.156965    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.156993    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jvc\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157099    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157135    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157178    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157206    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157271    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.157429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.157300    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.168027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.167998    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-eb6ef8-openshift-default-dockercfg-rxz8d\""
Nov 03 19:01:57.171838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.171810    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:01:57.258384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258352    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258427    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258474    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258543    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258572    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258620    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jvc\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258664    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258711    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258820    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.258979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.258875    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.259102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.259078    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.259162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.259116    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.259162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.259135    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.259325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.259300    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.261101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.261080    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.261440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.261415    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.315695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.315670    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.315864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.315848    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jvc\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.386479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.386444    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:57.524584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.524545    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"
Nov 03 19:01:57.524753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.524577    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr" event={"ID":"b48a47d9-4c6d-4b40-85d6-8c769c952843","Type":"ContainerDied","Data":"0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9"}
Nov 03 19:01:57.524753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.524637    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c20cfdbeb377f9679cfc545618a915ff1ebb0b0bc23e1b861d95af085f1ddf9"
Nov 03 19:01:57.818516 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:01:57.818486    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf168f41_4e76_4cc9_bf50_e2e9f0a7a125.slice/crio-6824344a9f706ed4dd79fea0ace86e047563c897fafc4997c6df13eb3b0333be WatchSource:0}: Error finding container 6824344a9f706ed4dd79fea0ace86e047563c897fafc4997c6df13eb3b0333be: Status 404 returned error can't find the container with id 6824344a9f706ed4dd79fea0ace86e047563c897fafc4997c6df13eb3b0333be
Nov 03 19:01:57.821689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.821653    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:57.821823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.821738    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:57.821897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.821817    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:01:57.846438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:57.846411    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:01:58.563242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:58.562118    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" event={"ID":"af168f41-4e76-4cc9-bf50-e2e9f0a7a125","Type":"ContainerStarted","Data":"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"}
Nov 03 19:01:58.563242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:58.562198    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" event={"ID":"af168f41-4e76-4cc9-bf50-e2e9f0a7a125","Type":"ContainerStarted","Data":"6824344a9f706ed4dd79fea0ace86e047563c897fafc4997c6df13eb3b0333be"}
Nov 03 19:01:59.386736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:59.386694    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:01:59.388291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:59.388267    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:01:59.388446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:01:59.388313    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:00.387227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:00.387190    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:02:00.387646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:00.387241    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:00.462259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:00.462227    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:01.387127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:01.387094    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:02:01.387313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:01.387145    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:02.387803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:02.387758    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:02:02.388180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:02.387827    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:03.387304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:03.387267    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:02:03.387475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:03.387323    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:04.387162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:04.387129    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 19:02:04.387565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:04.387182    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 19:02:04.703470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:04.703348    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:05.082557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:05.082514    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:05.330902 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:02:05.330868    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:02:05.391819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:05.391716    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:02:05.392277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:05.392126    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:02:05.392755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:05.392736    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:02:05.600001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:05.599947    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podStartSLOduration=8.5999348 podStartE2EDuration="8.5999348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:01:58.789080897 +0000 UTC m=+1017.066434492" watchObservedRunningTime="2026-11-03 19:02:05.5999348 +0000 UTC m=+1023.877288382"
Nov 03 19:02:08.150927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.150894    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"]
Nov 03 19:02:08.156250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.156230    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.205503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.205473    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:08.230245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.230213    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.230408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.230295    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4p48\" (UniqueName: \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.230470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.230395    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.251903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.251877    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:08.252016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.251905    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:08.252016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.251949    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:08.254417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.254398    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"]
Nov 03 19:02:08.331403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.331371    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4p48\" (UniqueName: \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.331403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.331409    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.331602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.331491    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.332068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.332048    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.333820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.333778    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.388458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.388422    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4p48\" (UniqueName: \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\") pod \"maas-api-revoke-sub-3cc8bf0a-7sfdj\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.469554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.469483    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:08.842625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:08.842595    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"]
Nov 03 19:02:08.848626 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:08.848597    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323d736a_94c4_4105_9cfa_bbafdb54dcd4.slice/crio-4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72 WatchSource:0}: Error finding container 4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72: Status 404 returned error can't find the container with id 4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72
Nov 03 19:02:09.632386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:09.632352    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj" event={"ID":"323d736a-94c4-4105-9cfa-bbafdb54dcd4","Type":"ContainerStarted","Data":"87c0beccab60542d685d3d440aa0940b1f02e60f9e53009b1ac0503ed7c99dad"}
Nov 03 19:02:09.632767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:09.632411    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj" event={"ID":"323d736a-94c4-4105-9cfa-bbafdb54dcd4","Type":"ContainerStarted","Data":"4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72"}
Nov 03 19:02:09.697576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:09.697453    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:09.742978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:09.742929    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj" podStartSLOduration=1.742916139 podStartE2EDuration="1.742916139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:09.740960546 +0000 UTC m=+1028.018314131" watchObservedRunningTime="2026-11-03 19:02:09.742916139 +0000 UTC m=+1028.020269723"
Nov 03 19:02:10.639112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:10.639077    4944 generic.go:356] "Generic (PLEG): container finished" podID="323d736a-94c4-4105-9cfa-bbafdb54dcd4" containerID="87c0beccab60542d685d3d440aa0940b1f02e60f9e53009b1ac0503ed7c99dad" exitCode=0
Nov 03 19:02:10.639498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:10.639158    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj" event={"ID":"323d736a-94c4-4105-9cfa-bbafdb54dcd4","Type":"ContainerDied","Data":"87c0beccab60542d685d3d440aa0940b1f02e60f9e53009b1ac0503ed7c99dad"}
Nov 03 19:02:11.139865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.139830    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:02:11.143759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.143740    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.242018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.241984    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-90a9bf-openshift-default-dockercfg-jqm4m\""
Nov 03 19:02:11.244586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.244565    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:02:11.266926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.266893    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267051    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267114    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267176    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267216    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r25cm\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267293    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267321    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267343    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.267452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.267391    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373032    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373092    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373238    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373337    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373493    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373733    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373830    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373860    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.373907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373893    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r25cm\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.374084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.373957    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.374294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.374272    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.374939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.374914    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.375698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.375306    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.375810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.375362    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.376462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.376440    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.377678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.377646    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.433059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.433001    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.436188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.436159    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r25cm\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.456917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.456896    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:11.779901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.779874    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:11.826443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.826414    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"]
Nov 03 19:02:11.827095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.827064    4944 cpu_manager.go:414] "containerMap: removing container" podUID="323d736a-94c4-4105-9cfa-bbafdb54dcd4" containerName="revoke-keys"
Nov 03 19:02:11.827095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.827087    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323d736a-94c4-4105-9cfa-bbafdb54dcd4" containerName="revoke-keys"
Nov 03 19:02:11.827285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.827272    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="323d736a-94c4-4105-9cfa-bbafdb54dcd4" containerName="revoke-keys"
Nov 03 19:02:11.830582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.830563    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.880461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.880429    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\") pod \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") "
Nov 03 19:02:11.880597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.880579    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\") pod \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") "
Nov 03 19:02:11.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.880744    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\" (UniqueName: \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\") pod \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\" (UID: \"323d736a-94c4-4105-9cfa-bbafdb54dcd4\") "
Nov 03 19:02:11.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.880889    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca" pod "323d736a-94c4-4105-9cfa-bbafdb54dcd4" (UID: "323d736a-94c4-4105-9cfa-bbafdb54dcd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:11.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.880963    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5mz\" (UniqueName: \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.881134    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.881248    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.883540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.881484    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:11.883540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.883389    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48" pod "323d736a-94c4-4105-9cfa-bbafdb54dcd4" (UID: "323d736a-94c4-4105-9cfa-bbafdb54dcd4"). InnerVolumeSpecName "kube-api-access-w4p48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:11.883701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.883653    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca" pod "323d736a-94c4-4105-9cfa-bbafdb54dcd4" (UID: "323d736a-94c4-4105-9cfa-bbafdb54dcd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:11.884309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.884287    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"]
Nov 03 19:02:11.891828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.891805    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:02:11.897777 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:11.897753    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0dd89a1_fd96_470c_b085_625fb3b12099.slice/crio-3d8ab134cf7c59e777ed117e97ca4707247377ebdf7d8f22aaa2b567a36d0157 WatchSource:0}: Error finding container 3d8ab134cf7c59e777ed117e97ca4707247377ebdf7d8f22aaa2b567a36d0157: Status 404 returned error can't find the container with id 3d8ab134cf7c59e777ed117e97ca4707247377ebdf7d8f22aaa2b567a36d0157
Nov 03 19:02:11.900388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.900360    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:11.900461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.900419    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:11.900461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.900447    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.983447    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.983680    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5mz\" (UniqueName: \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.983809    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.983881    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4p48\" (UniqueName: \"kubernetes.io/projected/323d736a-94c4-4105-9cfa-bbafdb54dcd4-kube-api-access-w4p48\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.983898    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323d736a-94c4-4105-9cfa-bbafdb54dcd4-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:11.985014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.984609    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:11.993808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:11.992144    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:12.043913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.043879    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5mz\" (UniqueName: \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\") pod \"maas-api-revoke-sub-372cc8ad-n722r\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:12.144093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.144060    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:12.514421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.514383    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"]
Nov 03 19:02:12.519889 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:12.519863    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd772ec7e_cf47_4717_a651_a767dd391911.slice/crio-30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7 WatchSource:0}: Error finding container 30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7: Status 404 returned error can't find the container with id 30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7
Nov 03 19:02:12.659491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.659393    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r" event={"ID":"d772ec7e-cf47-4717-a651-a767dd391911","Type":"ContainerStarted","Data":"30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7"}
Nov 03 19:02:12.660934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.660899    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"
Nov 03 19:02:12.660934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.660910    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj" event={"ID":"323d736a-94c4-4105-9cfa-bbafdb54dcd4","Type":"ContainerDied","Data":"4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72"}
Nov 03 19:02:12.661126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.660950    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c5dfa45c62f89d4946a07bffd986e82830a26e71fab0ca8a6c85d641cfa3e72"
Nov 03 19:02:12.662573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.662544    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" event={"ID":"b0dd89a1-fd96-470c-b085-625fb3b12099","Type":"ContainerStarted","Data":"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"}
Nov 03 19:02:12.662662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.662593    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" event={"ID":"b0dd89a1-fd96-470c-b085-625fb3b12099","Type":"ContainerStarted","Data":"3d8ab134cf7c59e777ed117e97ca4707247377ebdf7d8f22aaa2b567a36d0157"}
Nov 03 19:02:12.836567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:12.836497    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podStartSLOduration=1.836475498 podStartE2EDuration="1.836475498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:12.83560102 +0000 UTC m=+1031.112954640" watchObservedRunningTime="2026-11-03 19:02:12.836475498 +0000 UTC m=+1031.113829086"
Nov 03 19:02:13.458101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:13.458069    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:13.459383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:13.459363    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:13.459464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:13.459405    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:13.669968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:13.669939    4944 generic.go:356] "Generic (PLEG): container finished" podID="d772ec7e-cf47-4717-a651-a767dd391911" containerID="0a7315df14ca7dfa9d2110989ee2fe979a9bbad5f5bfcb31f30865756fd6ead2" exitCode=0
Nov 03 19:02:13.670110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:13.670012    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r" event={"ID":"d772ec7e-cf47-4717-a651-a767dd391911","Type":"ContainerDied","Data":"0a7315df14ca7dfa9d2110989ee2fe979a9bbad5f5bfcb31f30865756fd6ead2"}
Nov 03 19:02:14.276134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:14.276104    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:14.458274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:14.458237    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:14.458413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:14.458296    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:14.868253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:14.868225    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:15.039546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.039502    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\" (UniqueName: \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\") pod \"d772ec7e-cf47-4717-a651-a767dd391911\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") "
Nov 03 19:02:15.039757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.039722    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\") pod \"d772ec7e-cf47-4717-a651-a767dd391911\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") "
Nov 03 19:02:15.040375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.040349    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\") pod \"d772ec7e-cf47-4717-a651-a767dd391911\" (UID: \"d772ec7e-cf47-4717-a651-a767dd391911\") "
Nov 03 19:02:15.040904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.040878    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca" pod "d772ec7e-cf47-4717-a651-a767dd391911" (UID: "d772ec7e-cf47-4717-a651-a767dd391911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:15.041591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.041557    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:15.043319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.043295    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz" pod "d772ec7e-cf47-4717-a651-a767dd391911" (UID: "d772ec7e-cf47-4717-a651-a767dd391911"). InnerVolumeSpecName "kube-api-access-5d5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:15.044606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.044576    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca" pod "d772ec7e-cf47-4717-a651-a767dd391911" (UID: "d772ec7e-cf47-4717-a651-a767dd391911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:15.142841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.142805    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d772ec7e-cf47-4717-a651-a767dd391911-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:15.142841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.142838    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5mz\" (UniqueName: \"kubernetes.io/projected/d772ec7e-cf47-4717-a651-a767dd391911-kube-api-access-5d5mz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:15.457771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.457695    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:15.457771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.457753    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:15.692829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.691294    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r" event={"ID":"d772ec7e-cf47-4717-a651-a767dd391911","Type":"ContainerDied","Data":"30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7"}
Nov 03 19:02:15.692829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.691362    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30e554d0f6f686b08baeee507c7f5709edc0f3953a233de26cde2bd0e246b3d7"
Nov 03 19:02:15.692829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:15.691430    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"
Nov 03 19:02:16.467812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:16.467170    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:16.467812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:16.467235    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:17.458031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:17.457991    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:17.458240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:17.458055    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:18.458227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:18.458189    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 03 19:02:18.458715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:18.458253    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 03 19:02:19.369932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:19.369892    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:19.461425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:19.461397    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:19.461882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:19.461679    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:19.462689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:19.462666    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:02:21.388499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:21.388467    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:22.496121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.496087    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"]
Nov 03 19:02:22.496666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.496638    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d772ec7e-cf47-4717-a651-a767dd391911" containerName="revoke-keys"
Nov 03 19:02:22.496708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.496668    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d772ec7e-cf47-4717-a651-a767dd391911" containerName="revoke-keys"
Nov 03 19:02:22.496782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.496773    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d772ec7e-cf47-4717-a651-a767dd391911" containerName="revoke-keys"
Nov 03 19:02:22.500480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.500461    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.541960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.541928    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.542137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.542113    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxf8\" (UniqueName: \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.542189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.542175    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.554659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.554630    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"]
Nov 03 19:02:22.596970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.596912    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:22.596970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.596958    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:22.596970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.596963    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:22.597199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.597043    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:22.643022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.642989    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxf8\" (UniqueName: \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.643170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.643049    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.643170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.643097    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.643728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.643697    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.645379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.645352    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.700171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.700137    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxf8\" (UniqueName: \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\") pod \"maas-api-revoke-sub-3737e1b9-d7hkl\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:22.813248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:22.813161    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:23.185421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:23.185341    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"]
Nov 03 19:02:23.192305 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:23.192269    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8e07d8_cf94_4d5f_9bfc_8254266629cb.slice/crio-98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e WatchSource:0}: Error finding container 98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e: Status 404 returned error can't find the container with id 98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e
Nov 03 19:02:23.749814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:23.749747    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl" event={"ID":"7d8e07d8-cf94-4d5f-9bfc-8254266629cb","Type":"ContainerStarted","Data":"06c9def470bdcba9b3c41ffb53b737a8858f49b3638cfbc36e4704aef0e59b3a"}
Nov 03 19:02:23.750254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:23.749837    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl" event={"ID":"7d8e07d8-cf94-4d5f-9bfc-8254266629cb","Type":"ContainerStarted","Data":"98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e"}
Nov 03 19:02:23.857551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:23.857448    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl" podStartSLOduration=1.857434249 podStartE2EDuration="1.857434249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:23.856701188 +0000 UTC m=+1042.134054772" watchObservedRunningTime="2026-11-03 19:02:23.857434249 +0000 UTC m=+1042.134787850"
Nov 03 19:02:23.887399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:23.887365    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:24.757717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:24.757680    4944 generic.go:356] "Generic (PLEG): container finished" podID="7d8e07d8-cf94-4d5f-9bfc-8254266629cb" containerID="06c9def470bdcba9b3c41ffb53b737a8858f49b3638cfbc36e4704aef0e59b3a" exitCode=0
Nov 03 19:02:24.758162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:24.757768    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl" event={"ID":"7d8e07d8-cf94-4d5f-9bfc-8254266629cb","Type":"ContainerDied","Data":"06c9def470bdcba9b3c41ffb53b737a8858f49b3638cfbc36e4704aef0e59b3a"}
Nov 03 19:02:25.915687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.915656    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:25.977606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.977570    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\") pod \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") "
Nov 03 19:02:25.977775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.977638    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\" (UniqueName: \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\") pod \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") "
Nov 03 19:02:25.977775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.977767    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\") pod \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\" (UID: \"7d8e07d8-cf94-4d5f-9bfc-8254266629cb\") "
Nov 03 19:02:25.978028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.977997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca" pod "7d8e07d8-cf94-4d5f-9bfc-8254266629cb" (UID: "7d8e07d8-cf94-4d5f-9bfc-8254266629cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:25.978248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.978214    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:25.979732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.979709    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8" pod "7d8e07d8-cf94-4d5f-9bfc-8254266629cb" (UID: "7d8e07d8-cf94-4d5f-9bfc-8254266629cb"). InnerVolumeSpecName "kube-api-access-xrxf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:25.980235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:25.980215    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca" pod "7d8e07d8-cf94-4d5f-9bfc-8254266629cb" (UID: "7d8e07d8-cf94-4d5f-9bfc-8254266629cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:26.079817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:26.079724    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrxf8\" (UniqueName: \"kubernetes.io/projected/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-kube-api-access-xrxf8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:26.079817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:26.079753    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e07d8-cf94-4d5f-9bfc-8254266629cb-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:26.769878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:26.769843    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl" event={"ID":"7d8e07d8-cf94-4d5f-9bfc-8254266629cb","Type":"ContainerDied","Data":"98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e"}
Nov 03 19:02:26.770041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:26.769888    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"
Nov 03 19:02:26.770041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:26.769930    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f7c1df0229784776b0e6e11a04b76216e6d9726f95e4a6675e4e0ef332044e"
Nov 03 19:02:27.696204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.696170    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"]
Nov 03 19:02:27.696702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.696688    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7d8e07d8-cf94-4d5f-9bfc-8254266629cb" containerName="revoke-keys"
Nov 03 19:02:27.696746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.696703    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8e07d8-cf94-4d5f-9bfc-8254266629cb" containerName="revoke-keys"
Nov 03 19:02:27.696821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.696806    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8e07d8-cf94-4d5f-9bfc-8254266629cb" containerName="revoke-keys"
Nov 03 19:02:27.700389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.700371    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.754772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.754747    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:27.797176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.797142    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqv7b\" (UniqueName: \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.797347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.797196    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.797449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.797420    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.801508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.801482    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:27.801508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.801502    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:27.801715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.801701    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:27.806310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.806288    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"]
Nov 03 19:02:27.898177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.898140    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqv7b\" (UniqueName: \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.898347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.898200    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.898347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.898338    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.898985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.898965    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.900444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.900425    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:27.955218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:27.955145    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqv7b\" (UniqueName: \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\") pod \"maas-api-revoke-sub-b87a4115-2mgr7\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:28.013929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:28.013897    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:28.360015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:28.359979    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:28.385500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:28.385466    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"]
Nov 03 19:02:28.391067 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:28.391044    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2919be_f379_4be9_b7ba_3f2e209acd13.slice/crio-72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117 WatchSource:0}: Error finding container 72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117: Status 404 returned error can't find the container with id 72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117
Nov 03 19:02:28.785398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:28.785364    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7" event={"ID":"8a2919be-f379-4be9-b7ba-3f2e209acd13","Type":"ContainerStarted","Data":"72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117"}
Nov 03 19:02:29.792558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:29.792521    4944 generic.go:356] "Generic (PLEG): container finished" podID="8a2919be-f379-4be9-b7ba-3f2e209acd13" containerID="8c6556c0cda17a27e701573df84b8c56ed79b40a41eb5689f7a21166f3463a44" exitCode=0
Nov 03 19:02:29.792981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:29.792602    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7" event={"ID":"8a2919be-f379-4be9-b7ba-3f2e209acd13","Type":"ContainerDied","Data":"8c6556c0cda17a27e701573df84b8c56ed79b40a41eb5689f7a21166f3463a44"}
Nov 03 19:02:30.960257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:30.960236    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:31.043285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.040645    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\") pod \"8a2919be-f379-4be9-b7ba-3f2e209acd13\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") "
Nov 03 19:02:31.043285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.040774    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\") pod \"8a2919be-f379-4be9-b7ba-3f2e209acd13\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") "
Nov 03 19:02:31.043285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.041054    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\" (UniqueName: \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\") pod \"8a2919be-f379-4be9-b7ba-3f2e209acd13\" (UID: \"8a2919be-f379-4be9-b7ba-3f2e209acd13\") "
Nov 03 19:02:31.044255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.044208    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca" pod "8a2919be-f379-4be9-b7ba-3f2e209acd13" (UID: "8a2919be-f379-4be9-b7ba-3f2e209acd13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:31.059019 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.058949    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca" pod "8a2919be-f379-4be9-b7ba-3f2e209acd13" (UID: "8a2919be-f379-4be9-b7ba-3f2e209acd13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:31.060031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.060001    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b" pod "8a2919be-f379-4be9-b7ba-3f2e209acd13" (UID: "8a2919be-f379-4be9-b7ba-3f2e209acd13"). InnerVolumeSpecName "kube-api-access-cqv7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:31.142918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.142870    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqv7b\" (UniqueName: \"kubernetes.io/projected/8a2919be-f379-4be9-b7ba-3f2e209acd13-kube-api-access-cqv7b\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:31.142918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.142908    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:31.142918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.142919    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a2919be-f379-4be9-b7ba-3f2e209acd13-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:31.807640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.807601    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7" event={"ID":"8a2919be-f379-4be9-b7ba-3f2e209acd13","Type":"ContainerDied","Data":"72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117"}
Nov 03 19:02:31.807950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.807931    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72fb00033bab266c2716c3efde85383b56799c4d7b50481d5db36476a2778117"
Nov 03 19:02:31.808155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:31.808142    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"
Nov 03 19:02:33.076515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.076480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:33.530476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.530445    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"]
Nov 03 19:02:33.531051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.531034    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8a2919be-f379-4be9-b7ba-3f2e209acd13" containerName="revoke-keys"
Nov 03 19:02:33.531114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.531053    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2919be-f379-4be9-b7ba-3f2e209acd13" containerName="revoke-keys"
Nov 03 19:02:33.531164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.531153    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2919be-f379-4be9-b7ba-3f2e209acd13" containerName="revoke-keys"
Nov 03 19:02:33.535404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.535386    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.572255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.572226    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.572386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.572275    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzcl\" (UniqueName: \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.572454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.572405    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.591470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.591440    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"]
Nov 03 19:02:33.630376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.630340    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:33.630540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.630390    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:33.630540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.630433    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:33.630540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.630482    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:33.674083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.674054    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.674208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.674095    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzcl\" (UniqueName: \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.674208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.674136    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.674700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.674681    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.676465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.676434    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.730682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.730654    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzcl\" (UniqueName: \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\") pod \"maas-api-revoke-sub-caff934c-fxp9f\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:33.849131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:33.849055    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:34.248450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:34.248420    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"]
Nov 03 19:02:34.254735 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:34.254711    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97b1dac_a239_4ff0_a1a3_daa45766d41a.slice/crio-c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9 WatchSource:0}: Error finding container c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9: Status 404 returned error can't find the container with id c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9
Nov 03 19:02:34.826957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:34.826872    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f" event={"ID":"d97b1dac-a239-4ff0-a1a3-daa45766d41a","Type":"ContainerStarted","Data":"0d58303ec80f43cf7fcc28100f4d111fbf9d90e1965178f042721c7beee0045e"}
Nov 03 19:02:34.826957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:34.826920    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f" event={"ID":"d97b1dac-a239-4ff0-a1a3-daa45766d41a","Type":"ContainerStarted","Data":"c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9"}
Nov 03 19:02:34.939439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:34.939370    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f" podStartSLOduration=1.93935244 podStartE2EDuration="1.93935244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:34.938850593 +0000 UTC m=+1053.216204178" watchObservedRunningTime="2026-11-03 19:02:34.93935244 +0000 UTC m=+1053.216706023"
Nov 03 19:02:35.237519 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:02:35.237450    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:02:35.835979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:35.835943    4944 generic.go:356] "Generic (PLEG): container finished" podID="d97b1dac-a239-4ff0-a1a3-daa45766d41a" containerID="0d58303ec80f43cf7fcc28100f4d111fbf9d90e1965178f042721c7beee0045e" exitCode=0
Nov 03 19:02:35.836384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:35.836054    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f" event={"ID":"d97b1dac-a239-4ff0-a1a3-daa45766d41a","Type":"ContainerDied","Data":"0d58303ec80f43cf7fcc28100f4d111fbf9d90e1965178f042721c7beee0045e"}
Nov 03 19:02:36.993740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:36.993716    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:37.111425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.111346    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\" (UniqueName: \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\") pod \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") "
Nov 03 19:02:37.111597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.111467    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\") pod \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") "
Nov 03 19:02:37.111597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.111541    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\") pod \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\" (UID: \"d97b1dac-a239-4ff0-a1a3-daa45766d41a\") "
Nov 03 19:02:37.111915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.111892    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca" pod "d97b1dac-a239-4ff0-a1a3-daa45766d41a" (UID: "d97b1dac-a239-4ff0-a1a3-daa45766d41a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:37.112017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.112001    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:37.113571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.113550    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca" pod "d97b1dac-a239-4ff0-a1a3-daa45766d41a" (UID: "d97b1dac-a239-4ff0-a1a3-daa45766d41a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:37.113783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.113763    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl" pod "d97b1dac-a239-4ff0-a1a3-daa45766d41a" (UID: "d97b1dac-a239-4ff0-a1a3-daa45766d41a"). InnerVolumeSpecName "kube-api-access-dlzcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:37.212892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.212863    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzcl\" (UniqueName: \"kubernetes.io/projected/d97b1dac-a239-4ff0-a1a3-daa45766d41a-kube-api-access-dlzcl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:37.212892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.212890    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d97b1dac-a239-4ff0-a1a3-daa45766d41a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:37.851113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.851086    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"
Nov 03 19:02:37.851277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.851124    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f" event={"ID":"d97b1dac-a239-4ff0-a1a3-daa45766d41a","Type":"ContainerDied","Data":"c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9"}
Nov 03 19:02:37.851277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.851177    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3df084dd4ade11978aad6812352de074c3382c15389105816784982e15e58d9"
Nov 03 19:02:37.861370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:37.861334    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:38.294524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.294489    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"]
Nov 03 19:02:38.295073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.295057    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d97b1dac-a239-4ff0-a1a3-daa45766d41a" containerName="revoke-keys"
Nov 03 19:02:38.295073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.295073    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d97b1dac-a239-4ff0-a1a3-daa45766d41a" containerName="revoke-keys"
Nov 03 19:02:38.295180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.295157    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d97b1dac-a239-4ff0-a1a3-daa45766d41a" containerName="revoke-keys"
Nov 03 19:02:38.298771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.298751    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.353826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.353758    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"]
Nov 03 19:02:38.394875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.394841    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:38.394875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.394857    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:38.394875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.394841    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:38.395124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.394841    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:38.425105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.425070    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5782x\" (UniqueName: \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.425252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.425117    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.425252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.425236    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.526446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.526412    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5782x\" (UniqueName: \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.526651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.526459    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.526651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.526522    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.527324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.527295    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.529109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.529083    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.583361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.583284    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5782x\" (UniqueName: \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\") pod \"maas-api-revoke-sub-e237ec62-xwx55\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.611651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.611602    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:38.983249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:38.983174    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"]
Nov 03 19:02:38.989312 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:38.989282    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c3b539_f889_44cf_9dd5_e676eadc7427.slice/crio-d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e WatchSource:0}: Error finding container d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e: Status 404 returned error can't find the container with id d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e
Nov 03 19:02:39.864980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:39.864946    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55" event={"ID":"10c3b539-f889-44cf-9dd5-e676eadc7427","Type":"ContainerStarted","Data":"0ea1525706ea3330b7c1c7a1b6971c40cc5e971bdfeaefe96d5d6e481c7d49a3"}
Nov 03 19:02:39.865377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:39.865004    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55" event={"ID":"10c3b539-f889-44cf-9dd5-e676eadc7427","Type":"ContainerStarted","Data":"d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e"}
Nov 03 19:02:39.975056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:39.975002    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55" podStartSLOduration=1.974986963 podStartE2EDuration="1.974986963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:39.972103689 +0000 UTC m=+1058.249457272" watchObservedRunningTime="2026-11-03 19:02:39.974986963 +0000 UTC m=+1058.252340581"
Nov 03 19:02:40.872092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:40.872056    4944 generic.go:356] "Generic (PLEG): container finished" podID="10c3b539-f889-44cf-9dd5-e676eadc7427" containerID="0ea1525706ea3330b7c1c7a1b6971c40cc5e971bdfeaefe96d5d6e481c7d49a3" exitCode=0
Nov 03 19:02:40.872547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:40.872142    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55" event={"ID":"10c3b539-f889-44cf-9dd5-e676eadc7427","Type":"ContainerDied","Data":"0ea1525706ea3330b7c1c7a1b6971c40cc5e971bdfeaefe96d5d6e481c7d49a3"}
Nov 03 19:02:42.026211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.026183    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:42.168841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.168752    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\") pod \"10c3b539-f889-44cf-9dd5-e676eadc7427\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") "
Nov 03 19:02:42.168841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.168829    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\") pod \"10c3b539-f889-44cf-9dd5-e676eadc7427\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") "
Nov 03 19:02:42.169010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.168987    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\" (UniqueName: \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\") pod \"10c3b539-f889-44cf-9dd5-e676eadc7427\" (UID: \"10c3b539-f889-44cf-9dd5-e676eadc7427\") "
Nov 03 19:02:42.169216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.169195    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca" pod "10c3b539-f889-44cf-9dd5-e676eadc7427" (UID: "10c3b539-f889-44cf-9dd5-e676eadc7427"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:42.169378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.169357    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:42.170845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.170819    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca" pod "10c3b539-f889-44cf-9dd5-e676eadc7427" (UID: "10c3b539-f889-44cf-9dd5-e676eadc7427"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:42.170940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.170881    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x" pod "10c3b539-f889-44cf-9dd5-e676eadc7427" (UID: "10c3b539-f889-44cf-9dd5-e676eadc7427"). InnerVolumeSpecName "kube-api-access-5782x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:42.271264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.271217    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5782x\" (UniqueName: \"kubernetes.io/projected/10c3b539-f889-44cf-9dd5-e676eadc7427-kube-api-access-5782x\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:42.271264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.271258    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10c3b539-f889-44cf-9dd5-e676eadc7427-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:42.319534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.319430    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:42.463451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.463374    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"]
Nov 03 19:02:42.463958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.463943    4944 cpu_manager.go:414] "containerMap: removing container" podUID="10c3b539-f889-44cf-9dd5-e676eadc7427" containerName="revoke-keys"
Nov 03 19:02:42.464015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.463959    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c3b539-f889-44cf-9dd5-e676eadc7427" containerName="revoke-keys"
Nov 03 19:02:42.464094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.464085    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c3b539-f889-44cf-9dd5-e676eadc7427" containerName="revoke-keys"
Nov 03 19:02:42.470590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.470559    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.521224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.521197    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"]
Nov 03 19:02:42.565896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.565859    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:42.575585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.575556    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.575736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.575665    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.575822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.575743    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbgr\" (UniqueName: \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.676938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.676900    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbgr\" (UniqueName: \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.677089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.676981    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.677089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.677067    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.677761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.677741    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.679263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.679243    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.722420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.722362    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:02:42.726468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.726453    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.734244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.734222    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbgr\" (UniqueName: \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\") pod \"maas-api-revoke-sub-a2b59c18-vbtf5\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.775426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.775402    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-92596a-openshift-default-dockercfg-zf4cq\""
Nov 03 19:02:42.777925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.777900    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.777938    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.777970    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778098    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778153    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8m7\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778227    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778282    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778314    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.778432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.778416    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.784209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.784187    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:42.825202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.825166    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:02:42.879506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879469    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879527    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879578    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879659    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879688    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879724    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.879976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879902    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.880028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.879979    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8m7\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.880028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.880011    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.880458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.880429    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.881310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.881283    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.881886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.881865    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.882245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.882220    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.882385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.882368    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.883269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.883245    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.884650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.884623    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.888381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.888365    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"
Nov 03 19:02:42.888467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.888357    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55" event={"ID":"10c3b539-f889-44cf-9dd5-e676eadc7427","Type":"ContainerDied","Data":"d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e"}
Nov 03 19:02:42.888532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.888497    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64abd40e4843873c066a3dd766b0f1df1b91b6f6f8d4e2d67b49b1857fd145e"
Nov 03 19:02:42.937999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.937965    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:42.938749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:42.938731    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8m7\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:43.041101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.041068    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:43.159075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.159040    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"]
Nov 03 19:02:43.164701 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:43.164672    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164acba0_9da8_4c91_98cf_234fb7dc56eb.slice/crio-be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb WatchSource:0}: Error finding container be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb: Status 404 returned error can't find the container with id be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb
Nov 03 19:02:43.426679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.426656    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:02:43.431914 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:43.431892    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode771afd2_b988_461e_9626_2eb1bb142cd7.slice/crio-19547e1689ad81ac3f6c4cad13fc9cec7ea16f81be11faaa842f4a22a9358c9f WatchSource:0}: Error finding container 19547e1689ad81ac3f6c4cad13fc9cec7ea16f81be11faaa842f4a22a9358c9f: Status 404 returned error can't find the container with id 19547e1689ad81ac3f6c4cad13fc9cec7ea16f81be11faaa842f4a22a9358c9f
Nov 03 19:02:43.434216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.434190    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:43.434273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.434252    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:43.434304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.434283    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:02:43.466830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.466799    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"]
Nov 03 19:02:43.472272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.472255    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.534751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.533059    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"]
Nov 03 19:02:43.586723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.586693    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7cc4\" (UniqueName: \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.586920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.586827    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.586977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.586921    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.688224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.688146    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7cc4\" (UniqueName: \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.688370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.688269    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.688370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.688351    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.688928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.688908    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.690843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.690822    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.745554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.745519    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7cc4\" (UniqueName: \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\") pod \"maas-api-revoke-sub-ab58f010-4hcx6\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.789511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.789474    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:43.901213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.901140    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5" event={"ID":"164acba0-9da8-4c91-98cf-234fb7dc56eb","Type":"ContainerStarted","Data":"0aa3a4d023250f314ef2deece4619f7e54a6d11afa289aa9424f0041fd0c1f4b"}
Nov 03 19:02:43.901465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.901218    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5" event={"ID":"164acba0-9da8-4c91-98cf-234fb7dc56eb","Type":"ContainerStarted","Data":"be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb"}
Nov 03 19:02:43.905497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.905462    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" event={"ID":"e771afd2-b988-461e-9626-2eb1bb142cd7","Type":"ContainerStarted","Data":"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"}
Nov 03 19:02:43.905613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.905524    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" event={"ID":"e771afd2-b988-461e-9626-2eb1bb142cd7","Type":"ContainerStarted","Data":"19547e1689ad81ac3f6c4cad13fc9cec7ea16f81be11faaa842f4a22a9358c9f"}
Nov 03 19:02:43.936772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.936739    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 19:02:43.936993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:43.936967    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy" containerID="cri-o://ab17813fd02465dd682fab5ad5efc4f117b866edf111866b62c354dbb7aa0387" gracePeriod=30
Nov 03 19:02:44.030812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.023007    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5" podStartSLOduration=2.022985237 podStartE2EDuration="2.022985237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:44.018200329 +0000 UTC m=+1062.295553913" watchObservedRunningTime="2026-11-03 19:02:44.022985237 +0000 UTC m=+1062.300338822"
Nov 03 19:02:44.046823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.046130    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:44.048006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.047978    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:44.048131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.048030    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:44.137149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.136183    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podStartSLOduration=2.136160682 podStartE2EDuration="2.136160682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:44.133073146 +0000 UTC m=+1062.410426754" watchObservedRunningTime="2026-11-03 19:02:44.136160682 +0000 UTC m=+1062.413514266"
Nov 03 19:02:44.168094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.168063    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"]
Nov 03 19:02:44.173581 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:44.173458    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e665dc_3517_4704_9320_853689c8cf66.slice/crio-2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34 WatchSource:0}: Error finding container 2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34: Status 404 returned error can't find the container with id 2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34
Nov 03 19:02:44.953340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.953238    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6" event={"ID":"a7e665dc-3517-4704-9320-853689c8cf66","Type":"ContainerStarted","Data":"a5786b3a06533d5f824c4dbb2d4fb7ac17a3ce1d3bc0a99ee6a856106caf6238"}
Nov 03 19:02:44.953513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.953338    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6" event={"ID":"a7e665dc-3517-4704-9320-853689c8cf66","Type":"ContainerStarted","Data":"2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34"}
Nov 03 19:02:44.954875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.954839    4944 generic.go:356] "Generic (PLEG): container finished" podID="164acba0-9da8-4c91-98cf-234fb7dc56eb" containerID="0aa3a4d023250f314ef2deece4619f7e54a6d11afa289aa9424f0041fd0c1f4b" exitCode=0
Nov 03 19:02:44.954998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:44.954919    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5" event={"ID":"164acba0-9da8-4c91-98cf-234fb7dc56eb","Type":"ContainerDied","Data":"0aa3a4d023250f314ef2deece4619f7e54a6d11afa289aa9424f0041fd0c1f4b"}
Nov 03 19:02:45.042060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.042031    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:45.042271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.042082    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:45.063420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.063372    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6" podStartSLOduration=2.063359176 podStartE2EDuration="2.063359176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:45.060536199 +0000 UTC m=+1063.337889782" watchObservedRunningTime="2026-11-03 19:02:45.063359176 +0000 UTC m=+1063.340712794"
Nov 03 19:02:45.275190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.275154    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"]
Nov 03 19:02:45.279236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.279216    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.337321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.337285    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"]
Nov 03 19:02:45.435068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.435030    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9sq\" (UniqueName: \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.435237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.435115    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.435237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.435156    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.536918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.536841    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.537061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.536976    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9sq\" (UniqueName: \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.537061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.537053    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.537635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.537613    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.539281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.539251    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.593802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.593769    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9sq\" (UniqueName: \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\") pod \"maas-api-revoke-sub-50399228-q5q55\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.675771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.675742    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:45.963065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.962982    4944 generic.go:356] "Generic (PLEG): container finished" podID="a7e665dc-3517-4704-9320-853689c8cf66" containerID="a5786b3a06533d5f824c4dbb2d4fb7ac17a3ce1d3bc0a99ee6a856106caf6238" exitCode=0
Nov 03 19:02:45.963213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:45.963064    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6" event={"ID":"a7e665dc-3517-4704-9320-853689c8cf66","Type":"ContainerDied","Data":"a5786b3a06533d5f824c4dbb2d4fb7ac17a3ce1d3bc0a99ee6a856106caf6238"}
Nov 03 19:02:46.045083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.045047    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:46.045234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.045104    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:46.067746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.067712    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"]
Nov 03 19:02:46.072199 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:46.072168    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f8c208_2052_45e7_b0e8_34ccfcd8a502.slice/crio-d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160 WatchSource:0}: Error finding container d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160: Status 404 returned error can't find the container with id d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160
Nov 03 19:02:46.147059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.147033    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:46.245179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.245147    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\" (UniqueName: \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\") pod \"164acba0-9da8-4c91-98cf-234fb7dc56eb\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") "
Nov 03 19:02:46.245355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.245190    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\") pod \"164acba0-9da8-4c91-98cf-234fb7dc56eb\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") "
Nov 03 19:02:46.245433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.245388    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\") pod \"164acba0-9da8-4c91-98cf-234fb7dc56eb\" (UID: \"164acba0-9da8-4c91-98cf-234fb7dc56eb\") "
Nov 03 19:02:46.245739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.245709    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca" pod "164acba0-9da8-4c91-98cf-234fb7dc56eb" (UID: "164acba0-9da8-4c91-98cf-234fb7dc56eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:46.245949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.245931    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:46.247202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.247177    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca" pod "164acba0-9da8-4c91-98cf-234fb7dc56eb" (UID: "164acba0-9da8-4c91-98cf-234fb7dc56eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:46.247321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.247302    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr" pod "164acba0-9da8-4c91-98cf-234fb7dc56eb" (UID: "164acba0-9da8-4c91-98cf-234fb7dc56eb"). InnerVolumeSpecName "kube-api-access-gtbgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:46.347684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.347659    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtbgr\" (UniqueName: \"kubernetes.io/projected/164acba0-9da8-4c91-98cf-234fb7dc56eb-kube-api-access-gtbgr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:46.347778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.347686    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/164acba0-9da8-4c91-98cf-234fb7dc56eb-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:46.851223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.851145    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:46.971538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.971503    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"
Nov 03 19:02:46.971538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.971534    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5" event={"ID":"164acba0-9da8-4c91-98cf-234fb7dc56eb","Type":"ContainerDied","Data":"be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb"}
Nov 03 19:02:46.971821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.971582    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be1a25a70d75fb100c68fdd8ba05eab949d26d60e0c795c9e7d04166a7e388bb"
Nov 03 19:02:46.973222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.973191    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55" event={"ID":"c2f8c208-2052-45e7-b0e8-34ccfcd8a502","Type":"ContainerStarted","Data":"e162295ac7b14ac2aac04d98ba5d56f6606f1bf661f756fddb3ecdecc8521d0d"}
Nov 03 19:02:46.973356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:46.973244    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55" event={"ID":"c2f8c208-2052-45e7-b0e8-34ccfcd8a502","Type":"ContainerStarted","Data":"d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160"}
Nov 03 19:02:47.042204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.042162    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:47.042350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.042228    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:47.094730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.094664    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55" podStartSLOduration=2.094646633 podStartE2EDuration="2.094646633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:02:47.093570038 +0000 UTC m=+1065.370923620" watchObservedRunningTime="2026-11-03 19:02:47.094646633 +0000 UTC m=+1065.372000216"
Nov 03 19:02:47.162056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.162028    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:47.260094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.260061    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\" (UniqueName: \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\") pod \"a7e665dc-3517-4704-9320-853689c8cf66\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") "
Nov 03 19:02:47.260246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.260138    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\") pod \"a7e665dc-3517-4704-9320-853689c8cf66\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") "
Nov 03 19:02:47.260246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.260233    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\") pod \"a7e665dc-3517-4704-9320-853689c8cf66\" (UID: \"a7e665dc-3517-4704-9320-853689c8cf66\") "
Nov 03 19:02:47.260600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.260569    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca" pod "a7e665dc-3517-4704-9320-853689c8cf66" (UID: "a7e665dc-3517-4704-9320-853689c8cf66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:47.262096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.262061    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4" pod "a7e665dc-3517-4704-9320-853689c8cf66" (UID: "a7e665dc-3517-4704-9320-853689c8cf66"). InnerVolumeSpecName "kube-api-access-d7cc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:47.262188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.262165    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca" pod "a7e665dc-3517-4704-9320-853689c8cf66" (UID: "a7e665dc-3517-4704-9320-853689c8cf66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:47.362095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.362052    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7cc4\" (UniqueName: \"kubernetes.io/projected/a7e665dc-3517-4704-9320-853689c8cf66-kube-api-access-d7cc4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:47.362095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.362090    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:47.362338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.362106    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7e665dc-3517-4704-9320-853689c8cf66-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:47.979950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.979916    4944 generic.go:356] "Generic (PLEG): container finished" podID="c2f8c208-2052-45e7-b0e8-34ccfcd8a502" containerID="e162295ac7b14ac2aac04d98ba5d56f6606f1bf661f756fddb3ecdecc8521d0d" exitCode=0
Nov 03 19:02:47.980121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.979999    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55" event={"ID":"c2f8c208-2052-45e7-b0e8-34ccfcd8a502","Type":"ContainerDied","Data":"e162295ac7b14ac2aac04d98ba5d56f6606f1bf661f756fddb3ecdecc8521d0d"}
Nov 03 19:02:47.981215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.981197    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"
Nov 03 19:02:47.981328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.981218    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6" event={"ID":"a7e665dc-3517-4704-9320-853689c8cf66","Type":"ContainerDied","Data":"2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34"}
Nov 03 19:02:47.981328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:47.981262    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8bd897715aab8b0d3f24e9f1eee1618b5d046359092187a3895ab8a35e5d34"
Nov 03 19:02:48.042332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:48.042293    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:48.042476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:48.042349    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:48.988202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:48.988174    4944 generic.go:356] "Generic (PLEG): container finished" podID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerID="ab17813fd02465dd682fab5ad5efc4f117b866edf111866b62c354dbb7aa0387" exitCode=0
Nov 03 19:02:48.988510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:48.988245    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" event={"ID":"5e935542-6696-45c8-a529-21cabdd2b8ef","Type":"ContainerDied","Data":"ab17813fd02465dd682fab5ad5efc4f117b866edf111866b62c354dbb7aa0387"}
Nov 03 19:02:49.042208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.042176    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 03 19:02:49.042373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.042235    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 03 19:02:49.170920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.170893    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:49.284243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.284212    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\") pod \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") "
Nov 03 19:02:49.284440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.284382    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\") pod \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") "
Nov 03 19:02:49.284552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.284538    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\" (UniqueName: \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\") pod \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\" (UID: \"c2f8c208-2052-45e7-b0e8-34ccfcd8a502\") "
Nov 03 19:02:49.284719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.284698    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca" pod "c2f8c208-2052-45e7-b0e8-34ccfcd8a502" (UID: "c2f8c208-2052-45e7-b0e8-34ccfcd8a502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:49.284948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.284926    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.286286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.286265    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca" pod "c2f8c208-2052-45e7-b0e8-34ccfcd8a502" (UID: "c2f8c208-2052-45e7-b0e8-34ccfcd8a502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:02:49.286574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.286555    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq" pod "c2f8c208-2052-45e7-b0e8-34ccfcd8a502" (UID: "c2f8c208-2052-45e7-b0e8-34ccfcd8a502"). InnerVolumeSpecName "kube-api-access-zn9sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:49.386854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.386817    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9sq\" (UniqueName: \"kubernetes.io/projected/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-kube-api-access-zn9sq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.386854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.386853    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f8c208-2052-45e7-b0e8-34ccfcd8a502-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.601998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.601971    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 19:02:49.791494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791463    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.791643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791513    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.791643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791550    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.791756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791729    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.791859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791821    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.792005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791861    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.792005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791920    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.792005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791961    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.792194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.791996    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:02:49.792194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792012    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\") pod \"5e935542-6696-45c8-a529-21cabdd2b8ef\" (UID: \"5e935542-6696-45c8-a529-21cabdd2b8ef\") "
Nov 03 19:02:49.792194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792187    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:02:49.792503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792480    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:02:49.792596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792501    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:02:49.792655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792607    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.792655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792625    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.792655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792641    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.792832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792654    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.792953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.792928    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:02:49.794192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.794168    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:02:49.794297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.794196    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "kube-api-access-zpntq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:49.794297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.794244    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:02:49.794297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.794268    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token" pod "5e935542-6696-45c8-a529-21cabdd2b8ef" (UID: "5e935542-6696-45c8-a529-21cabdd2b8ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:02:49.894063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.893972    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.894063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.894012    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.894063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.894028    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e935542-6696-45c8-a529-21cabdd2b8ef-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.894063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.894042    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpntq\" (UniqueName: \"kubernetes.io/projected/5e935542-6696-45c8-a529-21cabdd2b8ef-kube-api-access-zpntq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.894063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.894059    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e935542-6696-45c8-a529-21cabdd2b8ef-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:02:49.996615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.996589    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"
Nov 03 19:02:49.997147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.996628    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55" event={"ID":"c2f8c208-2052-45e7-b0e8-34ccfcd8a502","Type":"ContainerDied","Data":"d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160"}
Nov 03 19:02:49.997147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.996688    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d40ed4c8d3b0cc239471173f05beffa7d8aaba1efaed25311eadf9f40e19c160"
Nov 03 19:02:49.998182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.998162    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"
Nov 03 19:02:49.998320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.998198    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s" event={"ID":"5e935542-6696-45c8-a529-21cabdd2b8ef","Type":"ContainerDied","Data":"1b0876d895cb33a8e18f2626c7116708f7e64c0f25ba4a1263e8e4d4f0073749"}
Nov 03 19:02:49.998320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:49.998254    4944 scope.go:122] "RemoveContainer" containerID="ab17813fd02465dd682fab5ad5efc4f117b866edf111866b62c354dbb7aa0387"
Nov 03 19:02:50.202692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:50.202593    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 19:02:50.250853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:50.250813    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4pb7s"]
Nov 03 19:02:50.338993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:50.338960    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:50.339950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:50.339924    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:50.340874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:50.340852    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:02:51.497624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:51.497592    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:51.755864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:51.755306    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:51.938448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:51.938408    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" path="/var/lib/kubelet/pods/5e935542-6696-45c8-a529-21cabdd2b8ef/volumes"
Nov 03 19:02:56.402996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:56.402963    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:02:58.379499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.379459    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"]
Nov 03 19:02:58.380231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380208    4944 cpu_manager.go:414] "containerMap: removing container" podUID="164acba0-9da8-4c91-98cf-234fb7dc56eb" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380232    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164acba0-9da8-4c91-98cf-234fb7dc56eb" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380253    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c2f8c208-2052-45e7-b0e8-34ccfcd8a502" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380261    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f8c208-2052-45e7-b0e8-34ccfcd8a502" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380275    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a7e665dc-3517-4704-9320-853689c8cf66" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380282    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e665dc-3517-4704-9320-853689c8cf66" containerName="revoke-keys"
Nov 03 19:02:58.380337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380333    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy"
Nov 03 19:02:58.380627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380340    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy"
Nov 03 19:02:58.380627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380476    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f8c208-2052-45e7-b0e8-34ccfcd8a502" containerName="revoke-keys"
Nov 03 19:02:58.380627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380488    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e665dc-3517-4704-9320-853689c8cf66" containerName="revoke-keys"
Nov 03 19:02:58.380627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380505    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e935542-6696-45c8-a529-21cabdd2b8ef" containerName="istio-proxy"
Nov 03 19:02:58.380627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.380517    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="164acba0-9da8-4c91-98cf-234fb7dc56eb" containerName="revoke-keys"
Nov 03 19:02:58.385509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.385488    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.425374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.425342    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.425529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.425398    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.425529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.425415    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmcr\" (UniqueName: \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.481344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.481306    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:02:58.481614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.481567    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:02:58.481747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.481721    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:02:58.481843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.481816    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:02:58.485074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.485052    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"]
Nov 03 19:02:58.527008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.526985    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.527100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.527052    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.527100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.527080    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmcr\" (UniqueName: \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.527710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.527689    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.529322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.529300    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.584042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.584016    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmcr\" (UniqueName: \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\") pod \"maas-api-revoke-sub-e8add3de-52hgw\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:58.698700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:58.698637    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:02:59.078242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:02:59.078206    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"]
Nov 03 19:02:59.083423 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:02:59.083379    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6873aa0_0e37_42a3_a0fe_37353d96edef.slice/crio-01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d WatchSource:0}: Error finding container 01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d: Status 404 returned error can't find the container with id 01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d
Nov 03 19:03:00.075681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:00.075646    4944 generic.go:356] "Generic (PLEG): container finished" podID="c6873aa0-0e37-42a3-a0fe-37353d96edef" containerID="34a3e8463514b07d825e22bebef56ead810c6f3821e9ca9d48ca4b70975aeff9" exitCode=0
Nov 03 19:03:00.076131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:00.075717    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw" event={"ID":"c6873aa0-0e37-42a3-a0fe-37353d96edef","Type":"ContainerDied","Data":"34a3e8463514b07d825e22bebef56ead810c6f3821e9ca9d48ca4b70975aeff9"}
Nov 03 19:03:00.076131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:00.075770    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw" event={"ID":"c6873aa0-0e37-42a3-a0fe-37353d96edef","Type":"ContainerStarted","Data":"01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d"}
Nov 03 19:03:01.009180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.009149    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:01.231385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.231354    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:03:01.363243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.363160    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\") pod \"c6873aa0-0e37-42a3-a0fe-37353d96edef\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") "
Nov 03 19:03:01.363399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.363256    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\" (UniqueName: \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\") pod \"c6873aa0-0e37-42a3-a0fe-37353d96edef\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") "
Nov 03 19:03:01.363487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.363460    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\") pod \"c6873aa0-0e37-42a3-a0fe-37353d96edef\" (UID: \"c6873aa0-0e37-42a3-a0fe-37353d96edef\") "
Nov 03 19:03:01.363601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.363504    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca" pod "c6873aa0-0e37-42a3-a0fe-37353d96edef" (UID: "c6873aa0-0e37-42a3-a0fe-37353d96edef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:01.363920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.363902    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:01.365394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.365361    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr" pod "c6873aa0-0e37-42a3-a0fe-37353d96edef" (UID: "c6873aa0-0e37-42a3-a0fe-37353d96edef"). InnerVolumeSpecName "kube-api-access-qlmcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:01.365495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.365393    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca" pod "c6873aa0-0e37-42a3-a0fe-37353d96edef" (UID: "c6873aa0-0e37-42a3-a0fe-37353d96edef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:01.464673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.464639    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6873aa0-0e37-42a3-a0fe-37353d96edef-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:01.464673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:01.464666    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlmcr\" (UniqueName: \"kubernetes.io/projected/c6873aa0-0e37-42a3-a0fe-37353d96edef-kube-api-access-qlmcr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:02.088647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:02.088617    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"
Nov 03 19:03:02.088861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:02.088650    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw" event={"ID":"c6873aa0-0e37-42a3-a0fe-37353d96edef","Type":"ContainerDied","Data":"01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d"}
Nov 03 19:03:02.088861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:02.088693    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c756035cfb6f5c9f2457e81e87aa11f2edd67b5890a35b6a687027fba25c3d"
Nov 03 19:03:05.224200 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:03:05.224154    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:03:05.661079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:05.661046    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:09.049442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.049398    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"]
Nov 03 19:03:09.050230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.050207    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c6873aa0-0e37-42a3-a0fe-37353d96edef" containerName="revoke-keys"
Nov 03 19:03:09.050337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.050232    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6873aa0-0e37-42a3-a0fe-37353d96edef" containerName="revoke-keys"
Nov 03 19:03:09.050393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.050349    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6873aa0-0e37-42a3-a0fe-37353d96edef" containerName="revoke-keys"
Nov 03 19:03:09.055189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.055167    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.104272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.104240    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:03:09.106462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.106435    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"]
Nov 03 19:03:09.143978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.143951    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.144110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.143983    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnz7c\" (UniqueName: \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.144155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.144115    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.148916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.148896    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:03:09.150367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.150349    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:03:09.150458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.150371    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:03:09.244775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.244747    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.244775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.244777    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnz7c\" (UniqueName: \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.244979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.244870    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.245390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.245369    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.247223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.247199    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.301545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.301489    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnz7c\" (UniqueName: \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\") pod \"maas-api-revoke-sub-8fb8f455-n5mff\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.368503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.368475    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:09.739818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:09.739782    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"]
Nov 03 19:03:09.745332 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:09.745287    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905ded08_5c6e_497d_b7c0_191d96f13421.slice/crio-bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3 WatchSource:0}: Error finding container bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3: Status 404 returned error can't find the container with id bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3
Nov 03 19:03:10.143344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:10.143306    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff" event={"ID":"905ded08-5c6e-497d-b7c0-191d96f13421","Type":"ContainerStarted","Data":"40d86faa80eafc841a4051c2d69063ea3740186858a52092fff5bf225f2b673d"}
Nov 03 19:03:10.143803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:10.143366    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff" event={"ID":"905ded08-5c6e-497d-b7c0-191d96f13421","Type":"ContainerStarted","Data":"bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3"}
Nov 03 19:03:10.183996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:10.183967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:10.251398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:10.251174    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff" podStartSLOduration=1.251159516 podStartE2EDuration="1.251159516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:10.250938531 +0000 UTC m=+1088.528292116" watchObservedRunningTime="2026-11-03 19:03:10.251159516 +0000 UTC m=+1088.528513099"
Nov 03 19:03:11.184086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.184050    4944 generic.go:356] "Generic (PLEG): container finished" podID="905ded08-5c6e-497d-b7c0-191d96f13421" containerID="40d86faa80eafc841a4051c2d69063ea3740186858a52092fff5bf225f2b673d" exitCode=0
Nov 03 19:03:11.184649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.184177    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff" event={"ID":"905ded08-5c6e-497d-b7c0-191d96f13421","Type":"ContainerDied","Data":"40d86faa80eafc841a4051c2d69063ea3740186858a52092fff5bf225f2b673d"}
Nov 03 19:03:11.309799 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.308891    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:03:11.314559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.313912    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.364526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.363688    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:03:11.485355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485255    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485416    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485482    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485634    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485699    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485771    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485844    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vptxj\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.485893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485876    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.486010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.485907    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.592614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.592575    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.592914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.592894    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vptxj\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593040    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593157    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593296    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593449    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593638    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593782    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.593946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.593933    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.594987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.594966    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.595432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.595415    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.597901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.596964    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.597901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.597211    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.597901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.597774    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.599174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.599155    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.601173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.601129    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.657317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.657282    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vptxj\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.658342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.658316    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:11.956817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:11.954292    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:12.385430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.385403    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:12.520899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.520858    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\") pod \"905ded08-5c6e-497d-b7c0-191d96f13421\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") "
Nov 03 19:03:12.521089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.520908    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\") pod \"905ded08-5c6e-497d-b7c0-191d96f13421\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") "
Nov 03 19:03:12.523243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.523209    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca" pod "905ded08-5c6e-497d-b7c0-191d96f13421" (UID: "905ded08-5c6e-497d-b7c0-191d96f13421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:12.523436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.523415    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\" (UniqueName: \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\") pod \"905ded08-5c6e-497d-b7c0-191d96f13421\" (UID: \"905ded08-5c6e-497d-b7c0-191d96f13421\") "
Nov 03 19:03:12.523983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.523953    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca" pod "905ded08-5c6e-497d-b7c0-191d96f13421" (UID: "905ded08-5c6e-497d-b7c0-191d96f13421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:12.524271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.524247    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:12.524390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.524376    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/905ded08-5c6e-497d-b7c0-191d96f13421-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:12.527247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.527223    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c" pod "905ded08-5c6e-497d-b7c0-191d96f13421" (UID: "905ded08-5c6e-497d-b7c0-191d96f13421"). InnerVolumeSpecName "kube-api-access-xnz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:12.545180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.545151    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:03:12.545999 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:12.545974    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae40ddfb_5a05_4863_beae_5c07be9dd146.slice/crio-6f849723f90aa6031be27b076ca621b8da006b21e41b23c51b14cfcedf27d40f WatchSource:0}: Error finding container 6f849723f90aa6031be27b076ca621b8da006b21e41b23c51b14cfcedf27d40f: Status 404 returned error can't find the container with id 6f849723f90aa6031be27b076ca621b8da006b21e41b23c51b14cfcedf27d40f
Nov 03 19:03:12.549780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.549742    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:12.549917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.549866    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:12.549991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.549917    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:12.625705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:12.625673    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnz7c\" (UniqueName: \"kubernetes.io/projected/905ded08-5c6e-497d-b7c0-191d96f13421-kube-api-access-xnz7c\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:13.199777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.199743    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"
Nov 03 19:03:13.199777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.199772    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff" event={"ID":"905ded08-5c6e-497d-b7c0-191d96f13421","Type":"ContainerDied","Data":"bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3"}
Nov 03 19:03:13.199990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.199834    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0fc8fd962c6b6a61381af01920c8ee2ad95a32ca73c9515c9d013a4f1dabc3"
Nov 03 19:03:13.201645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.201611    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" event={"ID":"ae40ddfb-5a05-4863-beae-5c07be9dd146","Type":"ContainerStarted","Data":"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"}
Nov 03 19:03:13.201818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.201652    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" event={"ID":"ae40ddfb-5a05-4863-beae-5c07be9dd146","Type":"ContainerStarted","Data":"6f849723f90aa6031be27b076ca621b8da006b21e41b23c51b14cfcedf27d40f"}
Nov 03 19:03:13.361208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.361145    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podStartSLOduration=2.361129438 podStartE2EDuration="2.361129438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:13.358159393 +0000 UTC m=+1091.635512976" watchObservedRunningTime="2026-11-03 19:03:13.361129438 +0000 UTC m=+1091.638483017"
Nov 03 19:03:13.955460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.955424    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:13.956911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.956883    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:13.957029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:13.956935    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:14.195639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.195604    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:03:14.196408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.196376    4944 cpu_manager.go:414] "containerMap: removing container" podUID="905ded08-5c6e-497d-b7c0-191d96f13421" containerName="revoke-keys"
Nov 03 19:03:14.196408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.196403    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="905ded08-5c6e-497d-b7c0-191d96f13421" containerName="revoke-keys"
Nov 03 19:03:14.196602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.196554    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="905ded08-5c6e-497d-b7c0-191d96f13421" containerName="revoke-keys"
Nov 03 19:03:14.200098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.200082    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.297941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.297911    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:03:14.347596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.347561    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.347777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.347621    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdsl\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.347917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.347816    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.347917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.347905    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.348023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.347970    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.348023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.348011    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.348163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.348145    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.348209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.348180    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.348271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.348257    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.449867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.449835    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdsl\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.449890    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.449939    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.449958    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.449981    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450178    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450226    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450258    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450320    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450344    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450407    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450599    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450739    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.450820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.450736    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.452426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.452399    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.452527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.452431    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.506649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.506623    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.520826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.506976    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdsl\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\") pod \"maas-default-gateway-openshift-default-77f556d647-tf8hc\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.538217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.538189    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:14.955831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.955717    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:14.955831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.955772    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:14.964255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:14.964224    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:03:15.006908 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:15.006867    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361411e7_0710_4f0f_8107_2cdd8c83af3b.slice/crio-daa0bdb6f39257d8d565db3fdada0be3d68ab9ecdb7907f7e8a93fffa772818d WatchSource:0}: Error finding container daa0bdb6f39257d8d565db3fdada0be3d68ab9ecdb7907f7e8a93fffa772818d: Status 404 returned error can't find the container with id daa0bdb6f39257d8d565db3fdada0be3d68ab9ecdb7907f7e8a93fffa772818d
Nov 03 19:03:15.033929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.033749    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:15.033929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.033881    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:15.034037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.033929    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:15.233054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.232864    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" event={"ID":"361411e7-0710-4f0f-8107-2cdd8c83af3b","Type":"ContainerStarted","Data":"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"}
Nov 03 19:03:15.233054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.232945    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" event={"ID":"361411e7-0710-4f0f-8107-2cdd8c83af3b","Type":"ContainerStarted","Data":"daa0bdb6f39257d8d565db3fdada0be3d68ab9ecdb7907f7e8a93fffa772818d"}
Nov 03 19:03:15.347867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.347782    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podStartSLOduration=1.347766829 podStartE2EDuration="1.347766829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:15.345390621 +0000 UTC m=+1093.622744205" watchObservedRunningTime="2026-11-03 19:03:15.347766829 +0000 UTC m=+1093.625120412"
Nov 03 19:03:15.475369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.475334    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:15.538977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.538942    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:15.958862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.958081    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:15.958862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:15.958157    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:16.541810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:16.541410    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:16.541810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:16.541487    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:16.957871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:16.956289    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:16.957871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:16.956370    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:17.539652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:17.539618    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:17.540245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:17.540215    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:17.957052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:17.956959    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:17.957309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:17.957280    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:18.546812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:18.542782    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:18.546812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:18.542882    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:18.955544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:18.955462    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 19:03:18.955544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:18.955513    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 19:03:19.540197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:19.540161    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:19.541245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:19.541212    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:19.959545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:19.959476    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:19.959954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:19.959903    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:19.960525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:19.960509    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:03:20.486661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:20.486628    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:20.539337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:20.539299    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:20.539507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:20.539364    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:21.539400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:21.539355    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:21.539812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:21.539435    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:21.740627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:21.740564    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:22.538904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:22.538872    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-tf8hc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 19:03:22.539081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:22.538922    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 19:03:23.596229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:23.596198    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:23.596879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:23.596857    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:23.597566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:23.597530    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:03:25.092457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:25.092426    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:26.139925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.139891    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:26.426193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.426111    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:03:26.429873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.429849    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514256    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514313    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514411    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514457    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514528    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514563    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514610    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4k2\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514719    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.514778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.514770    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.529124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.529097    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:03:26.615744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.615703    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.615933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.615757    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.615933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.615844    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4k2\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.615933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.615890    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.615939    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616014    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616073    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616111    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616158    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616166    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616324    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616357    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616403    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.616856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.616835    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.618341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.618314    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.618516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.618498    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.672724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.672701    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.672872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.672827    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4k2\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\") pod \"e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") " pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:26.742594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:26.742569    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:27.169287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.169202    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:03:27.174543 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:27.174513    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6625ab85_53ff_4163_ada7_c5c04cb6747b.slice/crio-3dc2eefd830800d365acfb56667f41e9422657bb0c123f20a7c886069a5207bc WatchSource:0}: Error finding container 3dc2eefd830800d365acfb56667f41e9422657bb0c123f20a7c886069a5207bc: Status 404 returned error can't find the container with id 3dc2eefd830800d365acfb56667f41e9422657bb0c123f20a7c886069a5207bc
Nov 03 19:03:27.176873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.176840    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:27.176948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.176910    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:27.176948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.176939    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:27.333688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.333646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" event={"ID":"6625ab85-53ff-4163-ada7-c5c04cb6747b","Type":"ContainerStarted","Data":"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"}
Nov 03 19:03:27.333922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.333710    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" event={"ID":"6625ab85-53ff-4163-ada7-c5c04cb6747b","Type":"ContainerStarted","Data":"3dc2eefd830800d365acfb56667f41e9422657bb0c123f20a7c886069a5207bc"}
Nov 03 19:03:27.452284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.452180    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podStartSLOduration=1.45216122 podStartE2EDuration="1.45216122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:27.448094668 +0000 UTC m=+1105.725448251" watchObservedRunningTime="2026-11-03 19:03:27.45216122 +0000 UTC m=+1105.729514803"
Nov 03 19:03:27.743055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:27.743021    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:28.744841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:28.744798    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:28.745228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:28.744878    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:29.298376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.298339    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:03:29.302820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.302780    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.358237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.358197    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:03:29.363217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.363192    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.364832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.364761    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365067    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365247    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365287    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365330    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365371    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9sdl\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365415    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365502    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.365668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.365594    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.391516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.391494    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:03:29.442098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.442062    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:03:29.466526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466487    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.466686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466568    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.466686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466601    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.466686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466635    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.466686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466668    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.466944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466701    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.466944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466844    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.466995    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467033    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxhlt\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467034    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467094    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9sdl\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467154    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467190    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467220    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467224    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467248    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467276    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467357    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.467623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467430    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467536    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467568    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.467623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.467575    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.468317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.468290    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.470179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.470153    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.470294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.470156    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.534486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.534454    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9sdl\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.534783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.534757    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6kcl\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.568731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568663    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.568731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568697    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.568731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568719    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568739    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568823    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxhlt\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568883    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568912    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568940    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.568998    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.569098    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.569224    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.569449    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.569546    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.569567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.569558    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.570830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.570813    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.571613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.571592    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.632279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.632247    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.633966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.633946    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxhlt\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\") pod \"maas-default-gateway-openshift-default-77f556d647-pd82h\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.657875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.657844    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:29.679409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.679380    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:29.744850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.743191    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:29.744850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.743269    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:29.750469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:29.750405    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:30.135169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.135140    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:03:30.136646 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:30.136611    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2ca6f6_e658_40bb_9a45_83ff8205c71e.slice/crio-75a3614e0663b2fa4f0603918a4ea590a83b97a90f92a7ba0030dbd9a76fad82 WatchSource:0}: Error finding container 75a3614e0663b2fa4f0603918a4ea590a83b97a90f92a7ba0030dbd9a76fad82: Status 404 returned error can't find the container with id 75a3614e0663b2fa4f0603918a4ea590a83b97a90f92a7ba0030dbd9a76fad82
Nov 03 19:03:30.139085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.139056    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.139161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.139117    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.139161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.139149    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.156397 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:30.156294    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4d849c_0681_411a_81c6_aef900ec09b2.slice/crio-6a99029fbaea16cc207c41170aed8d7e5b254ee40b5f988dcf2ba9b5ef9587f9 WatchSource:0}: Error finding container 6a99029fbaea16cc207c41170aed8d7e5b254ee40b5f988dcf2ba9b5ef9587f9: Status 404 returned error can't find the container with id 6a99029fbaea16cc207c41170aed8d7e5b254ee40b5f988dcf2ba9b5ef9587f9
Nov 03 19:03:30.159243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.159213    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.159335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.159285    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.159335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.159316    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:03:30.169904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.169879    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:03:30.370865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.370828    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" event={"ID":"8d4d849c-0681-411a-81c6-aef900ec09b2","Type":"ContainerStarted","Data":"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"}
Nov 03 19:03:30.371027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.370895    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" event={"ID":"8d4d849c-0681-411a-81c6-aef900ec09b2","Type":"ContainerStarted","Data":"6a99029fbaea16cc207c41170aed8d7e5b254ee40b5f988dcf2ba9b5ef9587f9"}
Nov 03 19:03:30.372928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.372889    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" event={"ID":"7e2ca6f6-e658-40bb-9a45-83ff8205c71e","Type":"ContainerStarted","Data":"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"}
Nov 03 19:03:30.373076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.372941    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" event={"ID":"7e2ca6f6-e658-40bb-9a45-83ff8205c71e","Type":"ContainerStarted","Data":"75a3614e0663b2fa4f0603918a4ea590a83b97a90f92a7ba0030dbd9a76fad82"}
Nov 03 19:03:30.491972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.491920    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podStartSLOduration=1.491905638 podStartE2EDuration="1.491905638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:30.490127204 +0000 UTC m=+1108.767480789" watchObservedRunningTime="2026-11-03 19:03:30.491905638 +0000 UTC m=+1108.769259221"
Nov 03 19:03:30.604506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.604439    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podStartSLOduration=1.604420669 podStartE2EDuration="1.604420669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:30.602565109 +0000 UTC m=+1108.879918729" watchObservedRunningTime="2026-11-03 19:03:30.604420669 +0000 UTC m=+1108.881774254"
Nov 03 19:03:30.658597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.658565    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:30.679612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.679586    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:30.743406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.743322    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:30.743406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:30.743383    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:31.661343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.660993    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:31.661343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.661056    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:31.683881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.683841    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:31.684191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.684131    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:31.759982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.759943    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:31.760269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:31.760240    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:32.672823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.671764    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:32.672823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.672700    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:32.683813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.681193    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:32.683813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.681248    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:32.749813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.749202    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:32.749813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:32.749264    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:33.663244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.663203    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:33.663526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.663497    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:33.680459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.680433    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:33.680886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.680864    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:33.743203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.743152    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 19:03:33.743470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:33.743443    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 19:03:34.669246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.669200    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:34.669424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.669267    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:34.680490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.679825    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:34.680490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.679868    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:34.723892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.723856    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:34.769041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.769006    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:34.769767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.769741    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:34.770727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:34.770708    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:03:35.273481 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:03:35.273441    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:03:35.658943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:35.658853    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:35.659210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:35.659179    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:35.679930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:35.679900    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:35.680076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:35.679948    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:36.659006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:36.658969    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:36.659377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:36.659034    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:36.679870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:36.679842    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:36.680008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:36.679891    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:37.658760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:37.658720    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6kcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Nov 03 19:03:37.658950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:37.658807    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Nov 03 19:03:37.680732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:37.680696    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pd82h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 03 19:03:37.681129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:37.680758    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 03 19:03:38.891108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.891072    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:38.891970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.891945    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:38.892874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.892847    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:03:38.991286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.990419    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:38.991286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.990826    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:38.991682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:38.991665    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:03:39.549735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:39.549707    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:44.097664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:44.097584    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:48.758689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:48.758656    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:50.896226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.896186    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"]
Nov 03 19:03:50.900361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.900337    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:50.996249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.996218    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:03:50.996424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.996371    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:03:50.996424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.996371    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:03:50.996516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.996437    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:03:50.999948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:50.999927    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"]
Nov 03 19:03:51.090919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.090887    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vldwd\" (UniqueName: \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.091099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.091083    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.091142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.091115    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.191971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.191896    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.191971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.191934    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.191971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.191966    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vldwd\" (UniqueName: \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.192515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.192496    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.194314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.194294    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.249341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.249315    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vldwd\" (UniqueName: \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\") pod \"maas-api-revoke-sub-f09170a2-s7pvl\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.514669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.514619    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:51.887488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.887411    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"]
Nov 03 19:03:51.892750 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:51.892718    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951f83b9_5c64_4fdd_aa6c_aa9fd53da75b.slice/crio-4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b WatchSource:0}: Error finding container 4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b: Status 404 returned error can't find the container with id 4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b
Nov 03 19:03:51.953639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.953606    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:03:51.959815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:51.959782    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.008997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.008876    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-92596a\"/\"distinct-w0-92596a-kserve-self-signed-certs\""
Nov 03 19:03:52.011668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.011634    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:03:52.081239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.081210    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"]
Nov 03 19:03:52.085282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.085263    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.105080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105054    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.105215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105109    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.105287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105234    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.105343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105326    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.105438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105423    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wcvd\" (UniqueName: \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.105500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.105465    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.140706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.140628    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"]
Nov 03 19:03:52.202096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.202062    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:52.206859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.206827    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.206999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.206951    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wcvd\" (UniqueName: \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.206999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.206986    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207018    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f6dh\" (UniqueName: \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.207126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207041    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.207126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207080    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207143    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207528    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207621    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207656    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.207777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.207731    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.208198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.208177    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.209982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.209961    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.210112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.210094    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.264146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.264118    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wcvd\" (UniqueName: \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\") pod \"distinct-w0-92596a-kserve-65484dfb85-4thw5\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") " pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.272575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.272553    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:52.309008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.308974    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f6dh\" (UniqueName: \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.309157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.309031    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.309263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.309248    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.309926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.309904    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.311480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.311461    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.380887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.380855    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f6dh\" (UniqueName: \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\") pod \"maas-api-revoke-sub-25c8719d-mhltc\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.398744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.398667    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:52.582990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.582953    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl" event={"ID":"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b","Type":"ContainerStarted","Data":"4d49d8bd1018f91ec1a7fba6341c8d5ab8afd77747ab0736edc2f5e8a11401bf"}
Nov 03 19:03:52.583133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.583007    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl" event={"ID":"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b","Type":"ContainerStarted","Data":"4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b"}
Nov 03 19:03:52.648319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.648283    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:03:52.653521 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:52.653493    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf635a375_39ee_4c80_914d_9819fab84b20.slice/crio-a9f38cab1b7c117bd74f73eaf2897dbc6adb4bb2e157fb6f6b00e15ec18bb38d WatchSource:0}: Error finding container a9f38cab1b7c117bd74f73eaf2897dbc6adb4bb2e157fb6f6b00e15ec18bb38d: Status 404 returned error can't find the container with id a9f38cab1b7c117bd74f73eaf2897dbc6adb4bb2e157fb6f6b00e15ec18bb38d
Nov 03 19:03:52.691246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.691087    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl" podStartSLOduration=2.691072134 podStartE2EDuration="2.691072134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:52.690559532 +0000 UTC m=+1130.967913115" watchObservedRunningTime="2026-11-03 19:03:52.691072134 +0000 UTC m=+1130.968425718"
Nov 03 19:03:52.785678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:52.785637    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"]
Nov 03 19:03:52.794379 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:52.794350    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0066318e_55be_486e_a19a_7791a7fe58ea.slice/crio-de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6 WatchSource:0}: Error finding container de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6: Status 404 returned error can't find the container with id de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6
Nov 03 19:03:53.376651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.376554    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:53.590687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.590653    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" event={"ID":"f635a375-39ee-4c80-914d-9819fab84b20","Type":"ContainerStarted","Data":"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"}
Nov 03 19:03:53.590919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.590698    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" event={"ID":"f635a375-39ee-4c80-914d-9819fab84b20","Type":"ContainerStarted","Data":"a9f38cab1b7c117bd74f73eaf2897dbc6adb4bb2e157fb6f6b00e15ec18bb38d"}
Nov 03 19:03:53.592162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.592136    4944 generic.go:356] "Generic (PLEG): container finished" podID="951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" containerID="4d49d8bd1018f91ec1a7fba6341c8d5ab8afd77747ab0736edc2f5e8a11401bf" exitCode=0
Nov 03 19:03:53.592265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.592216    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl" event={"ID":"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b","Type":"ContainerDied","Data":"4d49d8bd1018f91ec1a7fba6341c8d5ab8afd77747ab0736edc2f5e8a11401bf"}
Nov 03 19:03:53.593578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.593559    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc" event={"ID":"0066318e-55be-486e-a19a-7791a7fe58ea","Type":"ContainerStarted","Data":"c20b2af7f2e6ca030fd92a4ca2f0dc720466f441afa699646bb7ae0fe8e5aa24"}
Nov 03 19:03:53.593642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.593595    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc" event={"ID":"0066318e-55be-486e-a19a-7791a7fe58ea","Type":"ContainerStarted","Data":"de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6"}
Nov 03 19:03:53.702823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.701847    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podStartSLOduration=2.701831309 podStartE2EDuration="2.701831309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:53.701411544 +0000 UTC m=+1131.978765129" watchObservedRunningTime="2026-11-03 19:03:53.701831309 +0000 UTC m=+1131.979184896"
Nov 03 19:03:53.811359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:53.811303    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc" podStartSLOduration=1.811287246 podStartE2EDuration="1.811287246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:53.808671641 +0000 UTC m=+1132.086025224" watchObservedRunningTime="2026-11-03 19:03:53.811287246 +0000 UTC m=+1132.088640830"
Nov 03 19:03:54.273047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.272997    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:03:54.346102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.346064    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"]
Nov 03 19:03:54.350213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.350195    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.404636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.404607    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"]
Nov 03 19:03:54.436876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.436847    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.437019 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.436891    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgx2\" (UniqueName: \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.437061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.437038    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.538681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.538606    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.538681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.538646    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgx2\" (UniqueName: \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.538914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.538692    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.539255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.539232    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.540983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.540956    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.595582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.595555    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgx2\" (UniqueName: \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\") pod \"maas-api-revoke-sub-a4df9a87-6c2ks\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.602125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.602087    4944 generic.go:356] "Generic (PLEG): container finished" podID="0066318e-55be-486e-a19a-7791a7fe58ea" containerID="c20b2af7f2e6ca030fd92a4ca2f0dc720466f441afa699646bb7ae0fe8e5aa24" exitCode=0
Nov 03 19:03:54.602283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.602203    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc" event={"ID":"0066318e-55be-486e-a19a-7791a7fe58ea","Type":"ContainerDied","Data":"c20b2af7f2e6ca030fd92a4ca2f0dc720466f441afa699646bb7ae0fe8e5aa24"}
Nov 03 19:03:54.664636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.664405    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:54.769021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.768999    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:54.842676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.842584    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\" (UniqueName: \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\") pod \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") "
Nov 03 19:03:54.842868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.842701    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\") pod \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") "
Nov 03 19:03:54.842939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.842887    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\") pod \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\" (UID: \"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b\") "
Nov 03 19:03:54.843242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.843216    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca" pod "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" (UID: "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:54.843351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.843285    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:54.844623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.844594    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd" pod "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" (UID: "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b"). InnerVolumeSpecName "kube-api-access-vldwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:54.844731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.844695    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca" pod "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" (UID: "951f83b9-5c64-4fdd-aa6c-aa9fd53da75b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:54.944231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.944201    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vldwd\" (UniqueName: \"kubernetes.io/projected/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-kube-api-access-vldwd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:54.944231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:54.944230    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:55.054328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.054295    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"]
Nov 03 19:03:55.059326 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:55.059295    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3636feb3_201f_454b_992e_c82ef014d067.slice/crio-e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d WatchSource:0}: Error finding container e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d: Status 404 returned error can't find the container with id e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d
Nov 03 19:03:55.609705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.609675    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"
Nov 03 19:03:55.609705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.609701    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl" event={"ID":"951f83b9-5c64-4fdd-aa6c-aa9fd53da75b","Type":"ContainerDied","Data":"4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b"}
Nov 03 19:03:55.610209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.609743    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4168a6d0830eac04211f24b0f62e9af4ecee73a85e32b0d1eae9f5ed6ac5900b"
Nov 03 19:03:55.611655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.611630    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks" event={"ID":"3636feb3-201f-454b-992e-c82ef014d067","Type":"ContainerStarted","Data":"e6d9dde9683f9d44e1ed8c2b17cc7c55f99c9c0bf9bbcd212ad2f0706d1c04b8"}
Nov 03 19:03:55.611867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.611843    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks" event={"ID":"3636feb3-201f-454b-992e-c82ef014d067","Type":"ContainerStarted","Data":"e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d"}
Nov 03 19:03:55.722396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.722340    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks" podStartSLOduration=1.722321183 podStartE2EDuration="1.722321183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:55.719977056 +0000 UTC m=+1133.997330639" watchObservedRunningTime="2026-11-03 19:03:55.722321183 +0000 UTC m=+1133.999674767"
Nov 03 19:03:55.751245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.751220    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:55.853241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.853151    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\" (UniqueName: \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\") pod \"0066318e-55be-486e-a19a-7791a7fe58ea\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") "
Nov 03 19:03:55.853241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.853203    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\") pod \"0066318e-55be-486e-a19a-7791a7fe58ea\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") "
Nov 03 19:03:55.853465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.853331    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\") pod \"0066318e-55be-486e-a19a-7791a7fe58ea\" (UID: \"0066318e-55be-486e-a19a-7791a7fe58ea\") "
Nov 03 19:03:55.853588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.853560    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca" pod "0066318e-55be-486e-a19a-7791a7fe58ea" (UID: "0066318e-55be-486e-a19a-7791a7fe58ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:55.853774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.853758    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:55.855304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.855285    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh" pod "0066318e-55be-486e-a19a-7791a7fe58ea" (UID: "0066318e-55be-486e-a19a-7791a7fe58ea"). InnerVolumeSpecName "kube-api-access-4f6dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:55.855355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.855287    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca" pod "0066318e-55be-486e-a19a-7791a7fe58ea" (UID: "0066318e-55be-486e-a19a-7791a7fe58ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:55.954719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.954689    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f6dh\" (UniqueName: \"kubernetes.io/projected/0066318e-55be-486e-a19a-7791a7fe58ea-kube-api-access-4f6dh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:55.954862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:55.954734    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0066318e-55be-486e-a19a-7791a7fe58ea-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:56.620153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:56.620126    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"
Nov 03 19:03:56.620628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:56.620163    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc" event={"ID":"0066318e-55be-486e-a19a-7791a7fe58ea","Type":"ContainerDied","Data":"de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6"}
Nov 03 19:03:56.620628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:56.620212    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de9288f3708a8d3b838d3df9c0080bcde76a4c7aab202c3870952444753186e6"
Nov 03 19:03:56.621953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:56.621924    4944 generic.go:356] "Generic (PLEG): container finished" podID="3636feb3-201f-454b-992e-c82ef014d067" containerID="e6d9dde9683f9d44e1ed8c2b17cc7c55f99c9c0bf9bbcd212ad2f0706d1c04b8" exitCode=0
Nov 03 19:03:56.622075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:56.622019    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks" event={"ID":"3636feb3-201f-454b-992e-c82ef014d067","Type":"ContainerDied","Data":"e6d9dde9683f9d44e1ed8c2b17cc7c55f99c9c0bf9bbcd212ad2f0706d1c04b8"}
Nov 03 19:03:57.776736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.776714    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:57.876854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.876781    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\") pod \"3636feb3-201f-454b-992e-c82ef014d067\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") "
Nov 03 19:03:57.877034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.876989    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\" (UniqueName: \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\") pod \"3636feb3-201f-454b-992e-c82ef014d067\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") "
Nov 03 19:03:57.877122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.877102    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\") pod \"3636feb3-201f-454b-992e-c82ef014d067\" (UID: \"3636feb3-201f-454b-992e-c82ef014d067\") "
Nov 03 19:03:57.878263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.877414    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca" pod "3636feb3-201f-454b-992e-c82ef014d067" (UID: "3636feb3-201f-454b-992e-c82ef014d067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:03:57.878263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.877637    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636feb3-201f-454b-992e-c82ef014d067-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:57.878591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.878448    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:57.879442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.879373    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2" pod "3636feb3-201f-454b-992e-c82ef014d067" (UID: "3636feb3-201f-454b-992e-c82ef014d067"). InnerVolumeSpecName "kube-api-access-fqgx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:03:57.882929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.881401    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca" pod "3636feb3-201f-454b-992e-c82ef014d067" (UID: "3636feb3-201f-454b-992e-c82ef014d067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:03:57.978572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.978535    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgx2\" (UniqueName: \"kubernetes.io/projected/3636feb3-201f-454b-992e-c82ef014d067-kube-api-access-fqgx2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:57.978572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:57.978570    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636feb3-201f-454b-992e-c82ef014d067-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:03:58.287608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.287571    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:03:58.288236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288219    4944 cpu_manager.go:414] "containerMap: removing container" podUID="951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" containerName="revoke-keys"
Nov 03 19:03:58.288236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288236    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" containerName="revoke-keys"
Nov 03 19:03:58.288377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288250    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0066318e-55be-486e-a19a-7791a7fe58ea" containerName="revoke-keys"
Nov 03 19:03:58.288377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288256    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0066318e-55be-486e-a19a-7791a7fe58ea" containerName="revoke-keys"
Nov 03 19:03:58.288377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288265    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3636feb3-201f-454b-992e-c82ef014d067" containerName="revoke-keys"
Nov 03 19:03:58.288377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288270    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3636feb3-201f-454b-992e-c82ef014d067" containerName="revoke-keys"
Nov 03 19:03:58.288571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288385    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" containerName="revoke-keys"
Nov 03 19:03:58.288571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288393    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0066318e-55be-486e-a19a-7791a7fe58ea" containerName="revoke-keys"
Nov 03 19:03:58.288571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.288399    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3636feb3-201f-454b-992e-c82ef014d067" containerName="revoke-keys"
Nov 03 19:03:58.291954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.291938    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.346260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.346228    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:03:58.383514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383477    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.383678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383526    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.383678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383639    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.383752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383693    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlttr\" (UniqueName: \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.383825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383808    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.383870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.383847    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.387844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.387824    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"distinct-w1-010ed3-kserve-self-signed-certs\""
Nov 03 19:03:58.482228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.482174    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:03:58.485516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.485466    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.485968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.485945    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.485879    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486024    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486078    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlttr\" (UniqueName: \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486161    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486201    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486432    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.486455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.486520    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.488572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.488547    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.488888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.488868    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.543730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.543652    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlttr\" (UniqueName: \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\") pod \"distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.603988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.603957    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:03:58.637028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.636997    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks" event={"ID":"3636feb3-201f-454b-992e-c82ef014d067","Type":"ContainerDied","Data":"e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d"}
Nov 03 19:03:58.637227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.637052    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e411e06cfd4d7f741af14d1170245c34918a661432e90ebc7dbf22beecd41d0d"
Nov 03 19:03:58.637227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.637011    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"
Nov 03 19:03:58.983506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:58.983398    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:03:58.989657 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:03:58.989623    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ff60c1a_dbbf_41ad_b717_efa3eea672c2.slice/crio-e2a9a0f2fcd1e1b6ac1da7ebff667725fd46c2d6c6baa0a013f2b18fde11b4a1 WatchSource:0}: Error finding container e2a9a0f2fcd1e1b6ac1da7ebff667725fd46c2d6c6baa0a013f2b18fde11b4a1: Status 404 returned error can't find the container with id e2a9a0f2fcd1e1b6ac1da7ebff667725fd46c2d6c6baa0a013f2b18fde11b4a1
Nov 03 19:03:59.644966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:59.644933    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" event={"ID":"5ff60c1a-dbbf-41ad-b717-efa3eea672c2","Type":"ContainerStarted","Data":"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"}
Nov 03 19:03:59.645170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:59.644977    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" event={"ID":"5ff60c1a-dbbf-41ad-b717-efa3eea672c2","Type":"ContainerStarted","Data":"e2a9a0f2fcd1e1b6ac1da7ebff667725fd46c2d6c6baa0a013f2b18fde11b4a1"}
Nov 03 19:03:59.758567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:03:59.758496    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podStartSLOduration=1.758477936 podStartE2EDuration="1.758477936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:03:59.752305768 +0000 UTC m=+1138.029659350" watchObservedRunningTime="2026-11-03 19:03:59.758477936 +0000 UTC m=+1138.035831520"
Nov 03 19:04:00.605117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:00.605079    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:02.273754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:02.273720    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:04:02.286366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:02.286341    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:04:02.645681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:02.645597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:02.676760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:02.676731    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:04:04.914859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:04.914819    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"]
Nov 03 19:04:04.920569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:04.920547    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:04.973887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:04.973857    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"]
Nov 03 19:04:05.015150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.015114    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:04:05.015150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.015118    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:04:05.015362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.015124    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:04:05.015362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.015135    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:04:05.076078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.076042    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.076242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.076118    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d65dr\" (UniqueName: \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.076242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.076219    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.177569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.177488    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.177708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.177629    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.177847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.177823    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d65dr\" (UniqueName: \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.178252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.178231    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.179811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.179777    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.224467 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:04:05.224432    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:04:05.234690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.234665    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d65dr\" (UniqueName: \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\") pod \"maas-api-revoke-sub-a76c8f89-phl4j\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.234855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.234840    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:05.605193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.605155    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"]
Nov 03 19:04:05.609969 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:05.609943    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod347b2196_41c2_48e6_ba70_cc93dbe94cde.slice/crio-1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542 WatchSource:0}: Error finding container 1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542: Status 404 returned error can't find the container with id 1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542
Nov 03 19:04:05.688386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:05.688353    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j" event={"ID":"347b2196-41c2-48e6-ba70-cc93dbe94cde","Type":"ContainerStarted","Data":"1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542"}
Nov 03 19:04:06.590870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.590832    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"]
Nov 03 19:04:06.599295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.599263    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.650140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.650102    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"]
Nov 03 19:04:06.697247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.697216    4944 generic.go:356] "Generic (PLEG): container finished" podID="347b2196-41c2-48e6-ba70-cc93dbe94cde" containerID="e13e5d9e6adc1fb1b2e28f08974c22f922de2097ebbb30631bde00d98bb71ee2" exitCode=0
Nov 03 19:04:06.697428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.697259    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j" event={"ID":"347b2196-41c2-48e6-ba70-cc93dbe94cde","Type":"ContainerDied","Data":"e13e5d9e6adc1fb1b2e28f08974c22f922de2097ebbb30631bde00d98bb71ee2"}
Nov 03 19:04:06.701940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.701918    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8kn5\" (UniqueName: \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.702017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.701958    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.702017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.701998    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.803712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.803638    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8kn5\" (UniqueName: \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.803712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.803682    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.803942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.803722    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.804444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.804420    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.806210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.806189    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.861092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.861064    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8kn5\" (UniqueName: \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\") pod \"maas-api-revoke-sub-23c7149f-lpxhr\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:06.913758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:06.913729    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:07.206276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.206152    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:07.290699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.290640    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"]
Nov 03 19:04:07.295494 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:07.295463    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5ca783_df83_4149_946a_78b670d1f058.slice/crio-717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70 WatchSource:0}: Error finding container 717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70: Status 404 returned error can't find the container with id 717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70
Nov 03 19:04:07.352514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.352480    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"]
Nov 03 19:04:07.358073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.358053    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.411381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.411349    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"]
Nov 03 19:04:07.513477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.513450    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2kv4\" (UniqueName: \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.513629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.513492    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.513629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.513573    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.614817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.614760    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.615173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.614845    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.615173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.615022    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2kv4\" (UniqueName: \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.615601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.615579    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.616932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.616913    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.671744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.671719    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2kv4\" (UniqueName: \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\") pod \"maas-api-revoke-sub-5f7c2455-2x8f8\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:07.704758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.704725    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr" event={"ID":"8e5ca783-df83-4149-946a-78b670d1f058","Type":"ContainerStarted","Data":"4542d0e0a0f652b9f9c3c87d11c142c80bbf107ebbe0ddbf18303e29922ee62e"}
Nov 03 19:04:07.704894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.704778    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr" event={"ID":"8e5ca783-df83-4149-946a-78b670d1f058","Type":"ContainerStarted","Data":"717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70"}
Nov 03 19:04:07.816915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.816805    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr" podStartSLOduration=1.816776046 podStartE2EDuration="1.816776046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:07.811768346 +0000 UTC m=+1146.089121940" watchObservedRunningTime="2026-11-03 19:04:07.816776046 +0000 UTC m=+1146.094129628"
Nov 03 19:04:07.828305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.828282    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:07.918893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.918859    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\") pod \"347b2196-41c2-48e6-ba70-cc93dbe94cde\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") "
Nov 03 19:04:07.919045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.918922    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\") pod \"347b2196-41c2-48e6-ba70-cc93dbe94cde\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") "
Nov 03 19:04:07.919045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.919021    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\" (UniqueName: \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\") pod \"347b2196-41c2-48e6-ba70-cc93dbe94cde\" (UID: \"347b2196-41c2-48e6-ba70-cc93dbe94cde\") "
Nov 03 19:04:07.919310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.919280    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca" pod "347b2196-41c2-48e6-ba70-cc93dbe94cde" (UID: "347b2196-41c2-48e6-ba70-cc93dbe94cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:04:07.919457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.919439    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:07.920991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.920964    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca" pod "347b2196-41c2-48e6-ba70-cc93dbe94cde" (UID: "347b2196-41c2-48e6-ba70-cc93dbe94cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:04:07.921106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.920997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr" pod "347b2196-41c2-48e6-ba70-cc93dbe94cde" (UID: "347b2196-41c2-48e6-ba70-cc93dbe94cde"). InnerVolumeSpecName "kube-api-access-d65dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:07.970681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:07.970650    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:08.021047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.021008    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/347b2196-41c2-48e6-ba70-cc93dbe94cde-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:08.021047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.021045    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d65dr\" (UniqueName: \"kubernetes.io/projected/347b2196-41c2-48e6-ba70-cc93dbe94cde-kube-api-access-d65dr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:08.346422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.346370    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"]
Nov 03 19:04:08.350980 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:08.350955    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5697f688_855e_4b38_8639_7f1b840c352d.slice/crio-f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd WatchSource:0}: Error finding container f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd: Status 404 returned error can't find the container with id f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd
Nov 03 19:04:08.604955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.604925    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:08.617314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.617215    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:08.715001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.714969    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8" event={"ID":"5697f688-855e-4b38-8639-7f1b840c352d","Type":"ContainerStarted","Data":"b48419bfda2c58e6adbf146129f687660b88e3f79119e578c3c66cf63cee5dae"}
Nov 03 19:04:08.715162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.715022    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8" event={"ID":"5697f688-855e-4b38-8639-7f1b840c352d","Type":"ContainerStarted","Data":"f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd"}
Nov 03 19:04:08.716809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.716762    4944 generic.go:356] "Generic (PLEG): container finished" podID="8e5ca783-df83-4149-946a-78b670d1f058" containerID="4542d0e0a0f652b9f9c3c87d11c142c80bbf107ebbe0ddbf18303e29922ee62e" exitCode=0
Nov 03 19:04:08.716929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.716835    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr" event={"ID":"8e5ca783-df83-4149-946a-78b670d1f058","Type":"ContainerDied","Data":"4542d0e0a0f652b9f9c3c87d11c142c80bbf107ebbe0ddbf18303e29922ee62e"}
Nov 03 19:04:08.718367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.718347    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"
Nov 03 19:04:08.718493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.718382    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j" event={"ID":"347b2196-41c2-48e6-ba70-cc93dbe94cde","Type":"ContainerDied","Data":"1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542"}
Nov 03 19:04:08.718493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.718419    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2fc6fcbe5d54697bf3adc38420a9e3422dc89f4a4a3f4a267e52b2fd65c542"
Nov 03 19:04:08.732527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.732507    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:08.834743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:08.834634    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8" podStartSLOduration=1.834620521 podStartE2EDuration="1.834620521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:08.832738072 +0000 UTC m=+1147.110091667" watchObservedRunningTime="2026-11-03 19:04:08.834620521 +0000 UTC m=+1147.111974105"
Nov 03 19:04:09.730868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:09.730562    4944 generic.go:356] "Generic (PLEG): container finished" podID="5697f688-855e-4b38-8639-7f1b840c352d" containerID="b48419bfda2c58e6adbf146129f687660b88e3f79119e578c3c66cf63cee5dae" exitCode=0
Nov 03 19:04:09.730868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:09.730624    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8" event={"ID":"5697f688-855e-4b38-8639-7f1b840c352d","Type":"ContainerDied","Data":"b48419bfda2c58e6adbf146129f687660b88e3f79119e578c3c66cf63cee5dae"}
Nov 03 19:04:09.966114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:09.966082    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:10.171571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.168669    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\") pod \"8e5ca783-df83-4149-946a-78b670d1f058\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") "
Nov 03 19:04:10.171571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.168756    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\" (UniqueName: \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\") pod \"8e5ca783-df83-4149-946a-78b670d1f058\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") "
Nov 03 19:04:10.171903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.171874    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca" pod "8e5ca783-df83-4149-946a-78b670d1f058" (UID: "8e5ca783-df83-4149-946a-78b670d1f058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:04:10.172028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.171881    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\") pod \"8e5ca783-df83-4149-946a-78b670d1f058\" (UID: \"8e5ca783-df83-4149-946a-78b670d1f058\") "
Nov 03 19:04:10.172327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.172305    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:10.174009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.173984    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca" pod "8e5ca783-df83-4149-946a-78b670d1f058" (UID: "8e5ca783-df83-4149-946a-78b670d1f058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:04:10.174103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.174091    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5" pod "8e5ca783-df83-4149-946a-78b670d1f058" (UID: "8e5ca783-df83-4149-946a-78b670d1f058"). InnerVolumeSpecName "kube-api-access-d8kn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:10.273809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.273749    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5ca783-df83-4149-946a-78b670d1f058-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:10.273809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.273809    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8kn5\" (UniqueName: \"kubernetes.io/projected/8e5ca783-df83-4149-946a-78b670d1f058-kube-api-access-d8kn5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:10.739950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.739915    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr" event={"ID":"8e5ca783-df83-4149-946a-78b670d1f058","Type":"ContainerDied","Data":"717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70"}
Nov 03 19:04:10.739950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.739933    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"
Nov 03 19:04:10.740462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.739977    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="717427732f8f7dea21364b9c9c2ad04dd150ba546f645ecf9cc9bec6a8a9ac70"
Nov 03 19:04:10.975819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.974248    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:10.995503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.995411    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\") pod \"5697f688-855e-4b38-8639-7f1b840c352d\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") "
Nov 03 19:04:10.995673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.995542    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\") pod \"5697f688-855e-4b38-8639-7f1b840c352d\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") "
Nov 03 19:04:10.995673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.995577    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\" (UniqueName: \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\") pod \"5697f688-855e-4b38-8639-7f1b840c352d\" (UID: \"5697f688-855e-4b38-8639-7f1b840c352d\") "
Nov 03 19:04:10.995809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.995745    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca" pod "5697f688-855e-4b38-8639-7f1b840c352d" (UID: "5697f688-855e-4b38-8639-7f1b840c352d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:04:10.996624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.996596    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:10.998239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.998214    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca" pod "5697f688-855e-4b38-8639-7f1b840c352d" (UID: "5697f688-855e-4b38-8639-7f1b840c352d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:04:10.998346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:10.998236    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4" pod "5697f688-855e-4b38-8639-7f1b840c352d" (UID: "5697f688-855e-4b38-8639-7f1b840c352d"). InnerVolumeSpecName "kube-api-access-j2kv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:11.099299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.098322    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2kv4\" (UniqueName: \"kubernetes.io/projected/5697f688-855e-4b38-8639-7f1b840c352d-kube-api-access-j2kv4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:11.099299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.098350    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5697f688-855e-4b38-8639-7f1b840c352d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:11.755298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.755245    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8" event={"ID":"5697f688-855e-4b38-8639-7f1b840c352d","Type":"ContainerDied","Data":"f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd"}
Nov 03 19:04:11.755767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.755387    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f42e1b36d37a43b08122be4ab19f8abb9f765de3ebba6d7c37a84f30d8e215bd"
Nov 03 19:04:11.755767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.755462    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"
Nov 03 19:04:11.783215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:11.783149    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:16.361004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:16.360971    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:21.569371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:21.569341    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:22.220875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.220837    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:22.520073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520041    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"]
Nov 03 19:04:22.520631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520617    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5697f688-855e-4b38-8639-7f1b840c352d" containerName="revoke-keys"
Nov 03 19:04:22.520694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520632    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5697f688-855e-4b38-8639-7f1b840c352d" containerName="revoke-keys"
Nov 03 19:04:22.520694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520666    4944 cpu_manager.go:414] "containerMap: removing container" podUID="347b2196-41c2-48e6-ba70-cc93dbe94cde" containerName="revoke-keys"
Nov 03 19:04:22.520694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520671    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="347b2196-41c2-48e6-ba70-cc93dbe94cde" containerName="revoke-keys"
Nov 03 19:04:22.520807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520695    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8e5ca783-df83-4149-946a-78b670d1f058" containerName="revoke-keys"
Nov 03 19:04:22.520807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520700    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5ca783-df83-4149-946a-78b670d1f058" containerName="revoke-keys"
Nov 03 19:04:22.520807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520778    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5697f688-855e-4b38-8639-7f1b840c352d" containerName="revoke-keys"
Nov 03 19:04:22.520807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520799    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5ca783-df83-4149-946a-78b670d1f058" containerName="revoke-keys"
Nov 03 19:04:22.520943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.520810    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="347b2196-41c2-48e6-ba70-cc93dbe94cde" containerName="revoke-keys"
Nov 03 19:04:22.524532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.524516    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.577894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.577865    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"]
Nov 03 19:04:22.619294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.619263    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:04:22.619497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.619280    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:04:22.619559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.619322    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:04:22.619559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.619406    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:04:22.667066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.667034    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.667198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.667084    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.667263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.667205    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98xb\" (UniqueName: \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.768751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.768718    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.768921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.768767    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.768921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.768869    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98xb\" (UniqueName: \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.769425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.769406    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.771125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.771076    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.825756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.825724    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98xb\" (UniqueName: \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\") pod \"maas-api-revoke-sub-7635a2d1-6787n\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:22.838639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:22.838616    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:23.041775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.041686    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:23.210013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.209969    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"]
Nov 03 19:04:23.214330 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:23.214298    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80325897_670f_4d63_9ec0_1db7ad17f7cd.slice/crio-ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6 WatchSource:0}: Error finding container ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6: Status 404 returned error can't find the container with id ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6
Nov 03 19:04:23.284353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.284325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:23.841258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.841228    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n" event={"ID":"80325897-670f-4d63-9ec0-1db7ad17f7cd","Type":"ContainerStarted","Data":"3b6abac728901cf4fe5c5351772a1974fc22310d78f1eaf95841dec5443813f2"}
Nov 03 19:04:23.841630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.841286    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n" event={"ID":"80325897-670f-4d63-9ec0-1db7ad17f7cd","Type":"ContainerStarted","Data":"ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6"}
Nov 03 19:04:23.953159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:23.953099    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n" podStartSLOduration=1.953083034 podStartE2EDuration="1.953083034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:23.950628116 +0000 UTC m=+1162.227981700" watchObservedRunningTime="2026-11-03 19:04:23.953083034 +0000 UTC m=+1162.230436625"
Nov 03 19:04:24.849845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:24.849811    4944 generic.go:356] "Generic (PLEG): container finished" podID="80325897-670f-4d63-9ec0-1db7ad17f7cd" containerID="3b6abac728901cf4fe5c5351772a1974fc22310d78f1eaf95841dec5443813f2" exitCode=0
Nov 03 19:04:24.850257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:24.849848    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n" event={"ID":"80325897-670f-4d63-9ec0-1db7ad17f7cd","Type":"ContainerDied","Data":"3b6abac728901cf4fe5c5351772a1974fc22310d78f1eaf95841dec5443813f2"}
Nov 03 19:04:26.011858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.011828    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:26.109118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.109021    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\") pod \"80325897-670f-4d63-9ec0-1db7ad17f7cd\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") "
Nov 03 19:04:26.109294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.109137    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\" (UniqueName: \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\") pod \"80325897-670f-4d63-9ec0-1db7ad17f7cd\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") "
Nov 03 19:04:26.109294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.109212    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\") pod \"80325897-670f-4d63-9ec0-1db7ad17f7cd\" (UID: \"80325897-670f-4d63-9ec0-1db7ad17f7cd\") "
Nov 03 19:04:26.109439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.109410    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca" pod "80325897-670f-4d63-9ec0-1db7ad17f7cd" (UID: "80325897-670f-4d63-9ec0-1db7ad17f7cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:04:26.109685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.109667    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:26.111245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.111218    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb" pod "80325897-670f-4d63-9ec0-1db7ad17f7cd" (UID: "80325897-670f-4d63-9ec0-1db7ad17f7cd"). InnerVolumeSpecName "kube-api-access-w98xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:04:26.111340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.111260    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca" pod "80325897-670f-4d63-9ec0-1db7ad17f7cd" (UID: "80325897-670f-4d63-9ec0-1db7ad17f7cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:04:26.210990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.210958    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98xb\" (UniqueName: \"kubernetes.io/projected/80325897-670f-4d63-9ec0-1db7ad17f7cd-kube-api-access-w98xb\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:26.210990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.210986    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80325897-670f-4d63-9ec0-1db7ad17f7cd-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:04:26.342203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.342172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:26.864244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.864214    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"
Nov 03 19:04:26.864431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.864255    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n" event={"ID":"80325897-670f-4d63-9ec0-1db7ad17f7cd","Type":"ContainerDied","Data":"ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6"}
Nov 03 19:04:26.864431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:26.864335    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab23faffdb2a00db0d1dd8edefd521b86475306060cc2a1d40ee054a463bf3b6"
Nov 03 19:04:30.431303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.431269    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:04:30.431897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.431882    4944 cpu_manager.go:414] "containerMap: removing container" podUID="80325897-670f-4d63-9ec0-1db7ad17f7cd" containerName="revoke-keys"
Nov 03 19:04:30.431952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.431899    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80325897-670f-4d63-9ec0-1db7ad17f7cd" containerName="revoke-keys"
Nov 03 19:04:30.432030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.432013    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="80325897-670f-4d63-9ec0-1db7ad17f7cd" containerName="revoke-keys"
Nov 03 19:04:30.436142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.436121    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.493025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.492976    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:04:30.531015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.530980    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-619731\"/\"distinct-w2-619731-kserve-self-signed-certs\""
Nov 03 19:04:30.560213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560185    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.560340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560251    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.560340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560327    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.560426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560350    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.560467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560455    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.560550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.560538    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjsg\" (UniqueName: \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.638741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.638671    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:30.661712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.661677    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjsg\" (UniqueName: \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.661930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.661770    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.661930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.661866    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.661930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.661920    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.662108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.661950    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.662108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.662027    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.662334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.662310    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.662766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.662581    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.662904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.662701    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.665314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.665289    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.666273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.666246    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.719070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.719001    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjsg\" (UniqueName: \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\") pod \"distinct-w2-619731-kserve-bd9668bf4-vkphr\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") " pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.748804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.748764    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:30.902544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:30.902511    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:31.123518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:31.123483    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:04:31.128108 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:31.128083    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a9b4d29_2199_48e3_924c_f0ea100ac0c0.slice/crio-7c5e0d35ece73d6e4f51acebe2704f14712d4a6b52e6ee2c0da5dac883bad9d3 WatchSource:0}: Error finding container 7c5e0d35ece73d6e4f51acebe2704f14712d4a6b52e6ee2c0da5dac883bad9d3: Status 404 returned error can't find the container with id 7c5e0d35ece73d6e4f51acebe2704f14712d4a6b52e6ee2c0da5dac883bad9d3
Nov 03 19:04:31.904199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:31.904155    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" event={"ID":"9a9b4d29-2199-48e3-924c-f0ea100ac0c0","Type":"ContainerStarted","Data":"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"}
Nov 03 19:04:31.904577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:31.904207    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" event={"ID":"9a9b4d29-2199-48e3-924c-f0ea100ac0c0","Type":"ContainerStarted","Data":"7c5e0d35ece73d6e4f51acebe2704f14712d4a6b52e6ee2c0da5dac883bad9d3"}
Nov 03 19:04:32.013255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:32.013199    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podStartSLOduration=2.013186982 podStartE2EDuration="2.013186982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:32.01261774 +0000 UTC m=+1170.289971324" watchObservedRunningTime="2026-11-03 19:04:32.013186982 +0000 UTC m=+1170.290540565"
Nov 03 19:04:32.749086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:32.749050    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:35.225245 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:04:35.225206    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:04:35.489735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:35.489711    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:37.119040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:37.119006    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:39.999021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:39.998981    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:40.749123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:40.749088    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:40.762739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:40.762709    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:40.982224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:40.982198    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:41.651554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.651518    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:04:41.657044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.657021    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697343    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697422    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697473    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697531    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697602    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697639    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697812    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697873    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mprwc\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.697959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.697927    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.757023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.756984    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:04:41.799399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799362    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799415    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mprwc\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799448    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799483    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799617    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799670    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799744    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799849    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.799934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.799898    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.800023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.800000    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.800236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.800094    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.800356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.800291    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.800356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.800331    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.800675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.800655    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.801762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.801737    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.802228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.802211    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.856463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.856436    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.856602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.856503    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mprwc\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:41.974551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:41.974466    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:42.285382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.285352    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:04:42.291253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.291232    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.315719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.315697    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:04:42.319761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.319743    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.348476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.348444    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:04:42.388000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.387971    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:04:42.408464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408421    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.408606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408479    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.408606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408587    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.408728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408655    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.408728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408696    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.408875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408743    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.408875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408782    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.408875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408859    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.409018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408894    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkptm\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.409018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.408954    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.409018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409005    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.409151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409079    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.409198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409167    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zc52\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.409246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409223    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.409294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409253    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.409365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409349    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.409418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409389    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.409464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.409435    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.450695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.450663    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:04:42.454682 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:42.454657    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87517ec_5657_4fa7_85d6_d587ef05ba72.slice/crio-8dd875f7c76f6e0ee63ad57dd17de7c0d65e1963296bb60065bc0bb76653f9e2 WatchSource:0}: Error finding container 8dd875f7c76f6e0ee63ad57dd17de7c0d65e1963296bb60065bc0bb76653f9e2: Status 404 returned error can't find the container with id 8dd875f7c76f6e0ee63ad57dd17de7c0d65e1963296bb60065bc0bb76653f9e2
Nov 03 19:04:42.456859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.456830    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:42.456926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.456894    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:42.456966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.456926    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:42.511041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511015    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511066    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zc52\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511107    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511132    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511213    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511244    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511277    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511326    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511365    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511408    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511539    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511661    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511411    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.511898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511843    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.511898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511872    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511920    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.511965    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512031    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512041    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512082    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512096    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkptm\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512112    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512185    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512226    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512409    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512667    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.512777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512678    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.512899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.512872    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.514323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.514287    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.514880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.514859    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.515051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.515034    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.515381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.515362    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.568684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.568596    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.568876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.568835    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zc52\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.569053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.569035    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkptm\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.569150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.569131    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\") pod \"e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") " pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.607176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.607143    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:42.633219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.633190    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:42.960251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.960168    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:04:42.965498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.965480    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:42.986058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.986028    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" event={"ID":"b87517ec-5657-4fa7-85d6-d587ef05ba72","Type":"ContainerStarted","Data":"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"}
Nov 03 19:04:42.986191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:42.986082    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" event={"ID":"b87517ec-5657-4fa7-85d6-d587ef05ba72","Type":"ContainerStarted","Data":"8dd875f7c76f6e0ee63ad57dd17de7c0d65e1963296bb60065bc0bb76653f9e2"}
Nov 03 19:04:43.020691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020544    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.020691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020621    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsbrg\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.020960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020711    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.020960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020771    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.020960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020843    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.020960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020888    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.021175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020966    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.021175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.020995    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.021175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.021108    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.022314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.022290    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:04:43.045155 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:43.045128    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e84a970_7ab9_472e_94f5_e512849cc6d7.slice/crio-ea65e4d1df3e3f4042f3f492779a6e849037f9c12d9912fe479e7d3fdb869dc5 WatchSource:0}: Error finding container ea65e4d1df3e3f4042f3f492779a6e849037f9c12d9912fe479e7d3fdb869dc5: Status 404 returned error can't find the container with id ea65e4d1df3e3f4042f3f492779a6e849037f9c12d9912fe479e7d3fdb869dc5
Nov 03 19:04:43.049090 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:43.049060    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f32810d_48e1_4428_a6bf_c4844a8bd71d.slice/crio-3ad6367d6231e547058706af8b7f690f8f44261c7ac9d73ca117e200a972eca7 WatchSource:0}: Error finding container 3ad6367d6231e547058706af8b7f690f8f44261c7ac9d73ca117e200a972eca7: Status 404 returned error can't find the container with id 3ad6367d6231e547058706af8b7f690f8f44261c7ac9d73ca117e200a972eca7
Nov 03 19:04:43.049090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.049069    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.049262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.049159    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.049262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.049204    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.050285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.050256    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.050355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.050338    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.050400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.050382    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.069479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.069450    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:04:43.075640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.075609    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:04:43.122240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122217    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122269    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122311    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsbrg\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122365    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122422    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122520    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122562    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122641    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.122717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122677    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.123048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.122999    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.123048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.123029    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.123048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.123067    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.123250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.123107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.123250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.123187    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.124835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.124750    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.125013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.124994    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.182807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.179707    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.182807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.182208    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsbrg\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.190808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.188901    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podStartSLOduration=2.188883822 podStartE2EDuration="2.188883822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:43.187845158 +0000 UTC m=+1181.465198741" watchObservedRunningTime="2026-11-03 19:04:43.188883822 +0000 UTC m=+1181.466237408"
Nov 03 19:04:43.280367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.280331    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:43.346984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.346235    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:04:43.363010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.359682    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.406294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.406251    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:04:43.427767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.427736    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.427767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.427777    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.427818    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nc5r\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.427997    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.428180    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.428225    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.428345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.428372    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.428487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.428400    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.535607    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.535674    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.535889    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.535947    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.535976    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.536101    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.536155    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.536184    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nc5r\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.536253    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.537563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.537047    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.540277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.539882    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.542137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.541266    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.542480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.542425    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.542869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.542850    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.545476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.545430    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.547165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.547126    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.598546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.598511    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nc5r\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.599463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.599437    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.666138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.666102    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:04:43.670088 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:04:43.670062    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbab8f6_be89_4a17_b9e7_be736c42fb0e.slice/crio-8696b71315df84a28c91636c61c13a4fe8d911694905c7d0aa6f714079061af0 WatchSource:0}: Error finding container 8696b71315df84a28c91636c61c13a4fe8d911694905c7d0aa6f714079061af0: Status 404 returned error can't find the container with id 8696b71315df84a28c91636c61c13a4fe8d911694905c7d0aa6f714079061af0
Nov 03 19:04:43.673037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.673006    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.673141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.673076    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.673141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.673111    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:43.685729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.685706    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:43.977809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.977501    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:43.977809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.977624    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:43.977809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:43.977667    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:44.045869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.045552    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" event={"ID":"5e84a970-7ab9-472e-94f5-e512849cc6d7","Type":"ContainerStarted","Data":"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"}
Nov 03 19:04:44.045869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.045621    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" event={"ID":"5e84a970-7ab9-472e-94f5-e512849cc6d7","Type":"ContainerStarted","Data":"ea65e4d1df3e3f4042f3f492779a6e849037f9c12d9912fe479e7d3fdb869dc5"}
Nov 03 19:04:44.068814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.057876    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" event={"ID":"7f32810d-48e1-4428-a6bf-c4844a8bd71d","Type":"ContainerStarted","Data":"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"}
Nov 03 19:04:44.068814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.057944    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" event={"ID":"7f32810d-48e1-4428-a6bf-c4844a8bd71d","Type":"ContainerStarted","Data":"3ad6367d6231e547058706af8b7f690f8f44261c7ac9d73ca117e200a972eca7"}
Nov 03 19:04:44.086814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.086309    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" event={"ID":"5fbab8f6-be89-4a17-b9e7-be736c42fb0e","Type":"ContainerStarted","Data":"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"}
Nov 03 19:04:44.086814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.086367    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" event={"ID":"5fbab8f6-be89-4a17-b9e7-be736c42fb0e","Type":"ContainerStarted","Data":"8696b71315df84a28c91636c61c13a4fe8d911694905c7d0aa6f714079061af0"}
Nov 03 19:04:44.163199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.163161    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:04:44.221720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.221670    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:44.221989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.221969    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:44.222135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.222117    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:04:44.342900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.334290    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:44.342900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.334365    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:44.382814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.353864    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:44.382814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.378616    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podStartSLOduration=2.378592573 podStartE2EDuration="2.378592573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:44.191606652 +0000 UTC m=+1182.468960237" watchObservedRunningTime="2026-11-03 19:04:44.378592573 +0000 UTC m=+1182.655946157"
Nov 03 19:04:44.500564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.493658    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podStartSLOduration=2.493635966 podStartE2EDuration="2.493635966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:44.404039678 +0000 UTC m=+1182.681393261" watchObservedRunningTime="2026-11-03 19:04:44.493635966 +0000 UTC m=+1182.770989550"
Nov 03 19:04:44.537817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.537652    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podStartSLOduration=2.5376284829999998 podStartE2EDuration="2.537628483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:44.521191108 +0000 UTC m=+1182.798544692" watchObservedRunningTime="2026-11-03 19:04:44.537628483 +0000 UTC m=+1182.814982067"
Nov 03 19:04:44.613399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.612857    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:44.613399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.612979    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:44.613399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.613034    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:44.640463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.639840    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:44.640463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.639916    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:44.640463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.640378    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:44.726827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.723357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:44.975239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.975144    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:44.977861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:44.975873    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:45.271811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.271081    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" event={"ID":"5b35487a-7313-4827-9384-563cc1670fd7","Type":"ContainerStarted","Data":"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"}
Nov 03 19:04:45.271811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.271143    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" event={"ID":"5b35487a-7313-4827-9384-563cc1670fd7","Type":"ContainerStarted","Data":"1cba3efe3ba1c64f02d4aa22fb5858c3a72c572e8feb41fdab2c852a79e8296a"}
Nov 03 19:04:45.303957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.303918    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:45.330477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.330434    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:45.625889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.612002    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:45.625889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.612070    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:45.637824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.637149    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:45.637824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.637206    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:45.689816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.689350    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:45.694391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.693904    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:45.694391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.693963    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:45.733812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.721603    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:45.733812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.732919    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:45.754813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.750877    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:04:45.842827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.833880    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:04:46.003812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.987770    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:04:46.003812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.998881    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:46.003812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:45.998929    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:46.086816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.086157    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.285874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.285451    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.291514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.291465    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:46.291798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.291756    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:46.322227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.322183    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.556878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.543011    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.610813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.609956    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:46.610813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.610032    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:46.643824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.643129    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:46.643824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.643190    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:46.694640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.694600    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:46.694930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.694899    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:46.738815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.735878    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:04:46.738815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.736331    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:46.738815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.738268    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.738815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.738622    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:04:46.760813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.751601    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.764812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.763925    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:04:46.855874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.854952    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podStartSLOduration=3.854929401 podStartE2EDuration="3.854929401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:04:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:04:45.408502917 +0000 UTC m=+1183.685856504" watchObservedRunningTime="2026-11-03 19:04:46.854929401 +0000 UTC m=+1185.132282985"
Nov 03 19:04:46.855874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.855866    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:46.867812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.856329    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:04:46.941815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:46.924891    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": context deadline exceeded"
Nov 03 19:04:47.126869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.017401    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:47.126869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.017962    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:47.176852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.155741    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:47.176852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.155826    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:47.216849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.208390    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:47.239585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.232744    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:04:47.301814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.291350    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:04:47.301814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.291601    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:04:47.397866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.378589    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:47.397866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.389060    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:04:47.419822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.407940    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:47.419822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.408007    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:47.757311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.757018    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.804462    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.813872    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.841285    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.841344    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.841434    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.841480    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.845095    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.845145    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.871025    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:04:47.984855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:47.885930    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:04:48.120878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.054744    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.183815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.119570    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.264875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.145347    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 03 19:04:48.264875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.145478    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:04:48.264875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.121552    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.264875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.218577    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.279176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.279125    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:48.321849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.283183    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.323353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.291750    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.377213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.361086    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:48.386813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.382890    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:04:48.637012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.636911    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.686833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.664189    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:48.686833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.664950    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.686833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.664991    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:48.689176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.689142    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.729993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.729945    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:48.775871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.775085    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.824244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.823001    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.835814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.824534    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.887991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.887873    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:48.975260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.975225    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:48.975557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:48.975510    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:49.190384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.190273    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:04:49.271813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.263895    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.271813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.264320    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.286938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.281160    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:49.286938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.281218    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:49.325226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.325173    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.391284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.391239    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.611225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.611187    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:49.611488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.611457    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:49.633237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.633193    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded"
Nov 03 19:04:49.634705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.634678    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:49.634901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.634877    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:49.709820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.692375    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:49.709820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.692442    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:49.724083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.724045    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:49.781938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.781885    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.824941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.824886    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:49.825515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.825486    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:04:49.913714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:49.913593    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.000963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.000919    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.001265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.001229    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:50.195211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.195094    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:50.264486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.264443    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.265566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.265535    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.284701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.284669    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.285049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.285019    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:50.337514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.337471    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.391896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.391843    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:50.629808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.616521    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.629808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.616580    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:50.642813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.634150    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.666813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.643062    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:50.691815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.688643    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.691815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.688707    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:50.784819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.784391    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:50.839412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.837834    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:50.839412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.837925    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.915432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.914967    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:04:50.975261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.975220    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 19:04:50.992835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:50.980909    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 19:04:51.204996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.204893    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:04:51.270990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.270939    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.294247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.274872    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.343318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.342878    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:04:51.371816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.296774    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:51.371816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.357627    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:51.411059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.411003    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.608820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.608237    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:51.608820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.608303    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:51.640254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.640213    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:51.642703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.642667    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:51.690814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.686499    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:51.690814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.686559    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:51.752925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.752873    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Nov 03 19:04:51.787034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.786984    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.845315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.845270    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.845914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.845540    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:51.926053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:51.925941    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:04:52.133972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.129314    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.195901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.194762    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.221703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.221656    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:04:52.295826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.295717    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.303845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.296288    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.307395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.307343    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:52.307603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.307569    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:52.346264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.345891    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:04:52.412178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.411896    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:04:52.612409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.612360    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 03 19:04:52.612759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.612686    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 03 19:04:52.636669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.636636    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 03 19:04:52.636948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.636915    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 03 19:04:52.687289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.687251    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:52.687567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.687534    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:52.728812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.724003    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.788811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.788091    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:52.789100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.789060    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.792825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.789596    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:52.810820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.805704    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:04:52.866868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.862938    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.904810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.903860    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:52.978826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.977511    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:04:52.991928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:52.985891    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:53.121592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.121504    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:04:53.138144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.126698    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:04:53.140811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.138673    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:04:53.239283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.231880    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:53.315733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.314921    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:04:53.321037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.319120    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Nov 03 19:04:53.321037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.319417    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 03 19:04:53.321037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.319451    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 03 19:04:53.472854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.472103    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:04:53.504140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.504115    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:04:53.548495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.547324    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:04:53.571843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.571337    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:04:53.574069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.573279    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:04:53.614870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.614836    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:53.616169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.616073    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:53.617188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.617167    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:04:53.653158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.653128    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:53.653946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.653923    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:53.656208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.656191    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:04:53.686098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.686068    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Nov 03 19:04:53.686229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.686117    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Nov 03 19:04:53.909813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:53.909640    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:54.331715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:54.331677    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:04:54.894761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:54.894730    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:54.895342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:54.895320    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:54.896135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:54.896103    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:04:55.556264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:55.556233    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:55.556673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:55.556654    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:55.557435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:55.557412    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:04:58.906131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:04:58.906091    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:01.687154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:01.687117    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:02.066986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:02.066952    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:05:02.069761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:02.069739    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:05:03.684619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:03.684590    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:05.346820 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:05:05.334147    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:05:08.270775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:08.270733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:12.753735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.753657    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:12.818844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.818812    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:05:12.824530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.824508    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900527    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900581    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900671    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6wm\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900731    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900848    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900898    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.900952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.900943    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.901022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.901010    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.901074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.901064    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:12.924090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:12.924059    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:05:13.002507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002463    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002612    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6wm\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002696    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002754    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002869    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002905    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.002976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.002944    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003006    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003126    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003234    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003261    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003506    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003526    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.003731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.003610    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.005271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.005220    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.005474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.005456    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.059995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.059954    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.060156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.060078    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6wm\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.138629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.138592    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:13.565962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.565929    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:05:13.570255 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:05:13.570226    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68eb852d_fb20_4b7a_a105_12c26a7bcb38.slice/crio-379663e38639d656a3dbe937b5917c4b12a12af67bbfc6c8019c1b2aee5f490b WatchSource:0}: Error finding container 379663e38639d656a3dbe937b5917c4b12a12af67bbfc6c8019c1b2aee5f490b: Status 404 returned error can't find the container with id 379663e38639d656a3dbe937b5917c4b12a12af67bbfc6c8019c1b2aee5f490b
Nov 03 19:05:13.572944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.572902    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:13.573051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.573002    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:13.573103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.573050    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:05:13.741120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.741069    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" event={"ID":"68eb852d-fb20-4b7a-a105-12c26a7bcb38","Type":"ContainerStarted","Data":"6ab447841868f6d2ae8e831a8c834609182f790dd5b21186ded3e7bd93e052a7"}
Nov 03 19:05:13.741290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.741147    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" event={"ID":"68eb852d-fb20-4b7a-a105-12c26a7bcb38","Type":"ContainerStarted","Data":"379663e38639d656a3dbe937b5917c4b12a12af67bbfc6c8019c1b2aee5f490b"}
Nov 03 19:05:13.854082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.853976    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podStartSLOduration=1.853961357 podStartE2EDuration="1.853961357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:13.853250873 +0000 UTC m=+1212.130604456" watchObservedRunningTime="2026-11-03 19:05:13.853961357 +0000 UTC m=+1212.131314939"
Nov 03 19:05:13.928029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:13.927977    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:14.139167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:14.139080    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:14.141005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:14.140978    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:14.141142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:14.141024    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:14.178952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:14.178918    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:15.139858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:15.139821    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:15.140250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:15.139878    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:16.139185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:16.139147    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:16.139393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:16.139217    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:17.139288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:17.139242    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:17.139842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:17.139307    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:17.315226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:17.315182    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:18.139730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:18.139696    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:18.140152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:18.139750    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:18.145544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:18.145517    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:18.389682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:18.389651    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:19.139558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:19.139523    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:19.139711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:19.139574    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:20.139255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:20.139220    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 19:05:20.139663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:20.139282    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 19:05:21.143164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:21.143131    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:21.143646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:21.143498    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:21.144082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:21.144060    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:05:21.503421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:21.503386    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:21.753633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:21.753550    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:22.000293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:22.000259    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:22.589301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:22.589260    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:26.268892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:26.268778    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:26.759831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:26.759760    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:31.633012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:31.632970    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:35.225240 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:05:35.225202    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:05:36.581423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:36.581381    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:41.005738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:41.005701    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:41.253708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:41.253671    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:45.999168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:45.999135    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:48.938330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:48.938296    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:05:48.942503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:48.942481    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.012445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.012413    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:05:49.037783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.037742    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-eb6ef8\"/\"distinct-w6-eb6ef8-kserve-self-signed-certs\""
Nov 03 19:05:49.072151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072115    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.072458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072429    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.072624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072594    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.072702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072638    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt6lv\" (UniqueName: \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.072768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072715    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.072853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.072783    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.131334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.131300    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:49.174331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174296    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174379    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174407    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt6lv\" (UniqueName: \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174452    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174495    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174549    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.174808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.174773    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.175148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.175089    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.175364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.175344    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.177462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.177439    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.178009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.177989    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.232163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.232098    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt6lv\" (UniqueName: \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\") pod \"distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") " pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.255460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.255434    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:49.659280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:49.659248    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:05:49.663776 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:05:49.663745    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aa9d4f2_522b_4fe2_abe5_79b1cdc0a55d.slice/crio-3895464f72661e802d5d427531d4b6481a146a38ceece69a7daeec2af9b0324b WatchSource:0}: Error finding container 3895464f72661e802d5d427531d4b6481a146a38ceece69a7daeec2af9b0324b: Status 404 returned error can't find the container with id 3895464f72661e802d5d427531d4b6481a146a38ceece69a7daeec2af9b0324b
Nov 03 19:05:50.021733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:50.021700    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" event={"ID":"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d","Type":"ContainerStarted","Data":"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"}
Nov 03 19:05:50.022164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:50.021744    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" event={"ID":"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d","Type":"ContainerStarted","Data":"3895464f72661e802d5d427531d4b6481a146a38ceece69a7daeec2af9b0324b"}
Nov 03 19:05:50.131953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:50.131901    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podStartSLOduration=2.131887378 podStartE2EDuration="2.131887378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:05:50.130174784 +0000 UTC m=+1248.407528367" watchObservedRunningTime="2026-11-03 19:05:50.131887378 +0000 UTC m=+1248.409240962"
Nov 03 19:05:50.256406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:50.256366    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:50.523684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:50.523635    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:55.038266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:55.038230    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:05:59.256623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:59.256581    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:59.269880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:59.269855    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:05:59.715783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:05:59.715697    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:00.110402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:00.110372    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:06:04.389168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:04.389117    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:05.224863 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:06:05.224829    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:06:09.059251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:09.059210    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:11.008352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.008319    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:06:11.012878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.012859    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.084166 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.084131    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:06:11.106312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106283    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.106481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106321    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.106481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106456    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqx7\" (UniqueName: \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.106596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106579    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.106643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106615    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.106739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.106681    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.107773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.107755    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-010ed3\"/\"distinct-2-w1-010ed3-kserve-self-signed-certs\""
Nov 03 19:06:11.201691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.201650    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:11.208648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.208615    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.208857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.208661    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.208857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.208761    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.209094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.209011    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.209562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.209539    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.209664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.209632    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqx7\" (UniqueName: \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.209972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.209151    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.209972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.209178    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.210193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.210050    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.211718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.211696    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.211848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.211716    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.267521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.267433    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqx7\" (UniqueName: \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\") pod \"distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") " pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.325462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.325435    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:11.703025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.702947    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:06:11.707908 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:11.707884    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b997992_121e_47c8_ac54_d849d78fe5f5.slice/crio-d9df37e06ca27b23c8b6b13181546b744c05c9b9f414f187198f7f522c214e28 WatchSource:0}: Error finding container d9df37e06ca27b23c8b6b13181546b744c05c9b9f414f187198f7f522c214e28: Status 404 returned error can't find the container with id d9df37e06ca27b23c8b6b13181546b744c05c9b9f414f187198f7f522c214e28
Nov 03 19:06:11.810181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.810144    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"]
Nov 03 19:06:11.814689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.814661    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:11.865020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.864982    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:06:11.865173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.865075    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:06:11.865173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.864991    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:06:11.870044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.870021    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"]
Nov 03 19:06:11.910608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.910583    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:06:11.921273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.921249    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:11.921386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.921369    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:11.921468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:11.921454    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztt4f\" (UniqueName: \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.023102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.023068    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.023572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.023186    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.023572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.023325    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztt4f\" (UniqueName: \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.023916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.023887    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.025728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.025710    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.058204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.058173    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"]
Nov 03 19:06:12.062185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.062165    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-858316dc-wghj6"]
Nov 03 19:06:12.081058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.081032    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztt4f\" (UniqueName: \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\") pod \"maas-api-revoke-sub-7fa09cd3-pmf6h\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.132140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.132116    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:12.199817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.199762    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" event={"ID":"9b997992-121e-47c8-ac54-d849d78fe5f5","Type":"ContainerStarted","Data":"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"}
Nov 03 19:06:12.199953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.199820    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" event={"ID":"9b997992-121e-47c8-ac54-d849d78fe5f5","Type":"ContainerStarted","Data":"d9df37e06ca27b23c8b6b13181546b744c05c9b9f414f187198f7f522c214e28"}
Nov 03 19:06:12.312218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.312125    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podStartSLOduration=2.312111587 podStartE2EDuration="2.312111587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:12.309421392 +0000 UTC m=+1270.586774974" watchObservedRunningTime="2026-11-03 19:06:12.312111587 +0000 UTC m=+1270.589465170"
Nov 03 19:06:12.326374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.326344    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:12.508932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:12.508439    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"]
Nov 03 19:06:12.521055 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:12.521022    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067b0d5c_9484_4cab_bbc8_e1fd3d27981a.slice/crio-e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e WatchSource:0}: Error finding container e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e: Status 404 returned error can't find the container with id e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e
Nov 03 19:06:13.208233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:13.208138    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h" event={"ID":"067b0d5c-9484-4cab-bbc8-e1fd3d27981a","Type":"ContainerStarted","Data":"9fc8f84fbebbd9116f452347f96053872df1e1bcdc237bcc24fc24c12898dbee"}
Nov 03 19:06:13.208233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:13.208200    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h" event={"ID":"067b0d5c-9484-4cab-bbc8-e1fd3d27981a","Type":"ContainerStarted","Data":"e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e"}
Nov 03 19:06:13.365390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:13.365327    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h" podStartSLOduration=2.365313501 podStartE2EDuration="2.365313501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:13.31645389 +0000 UTC m=+1271.593807473" watchObservedRunningTime="2026-11-03 19:06:13.365313501 +0000 UTC m=+1271.642667083"
Nov 03 19:06:13.683767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:13.683733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:13.934982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:13.934900    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aebb74d-54ce-4830-9271-1861c20a863a" path="/var/lib/kubelet/pods/7aebb74d-54ce-4830-9271-1861c20a863a/volumes"
Nov 03 19:06:14.216544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:14.216460    4944 generic.go:356] "Generic (PLEG): container finished" podID="067b0d5c-9484-4cab-bbc8-e1fd3d27981a" containerID="9fc8f84fbebbd9116f452347f96053872df1e1bcdc237bcc24fc24c12898dbee" exitCode=0
Nov 03 19:06:14.216544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:14.216506    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h" event={"ID":"067b0d5c-9484-4cab-bbc8-e1fd3d27981a","Type":"ContainerDied","Data":"9fc8f84fbebbd9116f452347f96053872df1e1bcdc237bcc24fc24c12898dbee"}
Nov 03 19:06:15.380492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.380465    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:15.470442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.470410    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\") pod \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") "
Nov 03 19:06:15.470676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.470663    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\") pod \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") "
Nov 03 19:06:15.470737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.470714    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\" (UniqueName: \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\") pod \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\" (UID: \"067b0d5c-9484-4cab-bbc8-e1fd3d27981a\") "
Nov 03 19:06:15.470833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.470776    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca" pod "067b0d5c-9484-4cab-bbc8-e1fd3d27981a" (UID: "067b0d5c-9484-4cab-bbc8-e1fd3d27981a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:06:15.471148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.471124    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:15.472612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.472587    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca" pod "067b0d5c-9484-4cab-bbc8-e1fd3d27981a" (UID: "067b0d5c-9484-4cab-bbc8-e1fd3d27981a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:06:15.472742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.472721    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f" pod "067b0d5c-9484-4cab-bbc8-e1fd3d27981a" (UID: "067b0d5c-9484-4cab-bbc8-e1fd3d27981a"). InnerVolumeSpecName "kube-api-access-ztt4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:15.572393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.572360    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:15.572393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:15.572390    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztt4f\" (UniqueName: \"kubernetes.io/projected/067b0d5c-9484-4cab-bbc8-e1fd3d27981a-kube-api-access-ztt4f\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:16.231385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:16.231301    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"
Nov 03 19:06:16.231385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:16.231338    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h" event={"ID":"067b0d5c-9484-4cab-bbc8-e1fd3d27981a","Type":"ContainerDied","Data":"e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e"}
Nov 03 19:06:16.231385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:16.231383    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e415d5056c7de468117c08f7d1666a19315e4f820abf0c444dc1b151f100f83e"
Nov 03 19:06:18.363227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:18.363181    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:21.326324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:21.326293    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:21.338864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:21.338840    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:22.291282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:22.291251    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:06:22.882821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:22.882766    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:23.128531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:23.128493    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:26.065820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.064611    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"]
Nov 03 19:06:26.065820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.065677    4944 cpu_manager.go:414] "containerMap: removing container" podUID="067b0d5c-9484-4cab-bbc8-e1fd3d27981a" containerName="revoke-keys"
Nov 03 19:06:26.065820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.065701    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067b0d5c-9484-4cab-bbc8-e1fd3d27981a" containerName="revoke-keys"
Nov 03 19:06:26.066404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.065902    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="067b0d5c-9484-4cab-bbc8-e1fd3d27981a" containerName="revoke-keys"
Nov 03 19:06:26.077905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.074217    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.116817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.116417    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"]
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.321430    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.352966    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.364271    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.364314    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srgw\" (UniqueName: \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.364484    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.390951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.379507    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:06:26.627626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.627567    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.699365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.699266    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.716426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.714149    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srgw\" (UniqueName: \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.716426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.647258    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:06:26.716426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.714037    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.807811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.801352    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.849809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.838169    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srgw\" (UniqueName: \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\") pod \"maas-api-revoke-sub-424ed2af-kzmtj\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:26.921348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:26.921304    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:06:27.031815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.031115    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:27.367282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.367197    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:27.465273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.465238    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"]
Nov 03 19:06:27.469854 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:27.469825    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4960f29_fa8c_4514_80ff_9f927d04ea94.slice/crio-1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997 WatchSource:0}: Error finding container 1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997: Status 404 returned error can't find the container with id 1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997
Nov 03 19:06:27.612903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.612871    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:27.893506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.893415    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj" event={"ID":"f4960f29-fa8c-4514-80ff-9f927d04ea94","Type":"ContainerStarted","Data":"de69ac18f9b25ea9728eaf2a2bca8f03829d125d7690bc1b5e95fe4df0912e0f"}
Nov 03 19:06:27.893506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.893488    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj" event={"ID":"f4960f29-fa8c-4514-80ff-9f927d04ea94","Type":"ContainerStarted","Data":"1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997"}
Nov 03 19:06:27.979298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:27.979262    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:28.006002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:28.005949    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj" podStartSLOduration=3.005934887 podStartE2EDuration="3.005934887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:28.001171426 +0000 UTC m=+1286.278525025" watchObservedRunningTime="2026-11-03 19:06:28.005934887 +0000 UTC m=+1286.283288496"
Nov 03 19:06:28.903247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:28.903213    4944 generic.go:356] "Generic (PLEG): container finished" podID="f4960f29-fa8c-4514-80ff-9f927d04ea94" containerID="de69ac18f9b25ea9728eaf2a2bca8f03829d125d7690bc1b5e95fe4df0912e0f" exitCode=0
Nov 03 19:06:28.903633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:28.903288    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj" event={"ID":"f4960f29-fa8c-4514-80ff-9f927d04ea94","Type":"ContainerDied","Data":"de69ac18f9b25ea9728eaf2a2bca8f03829d125d7690bc1b5e95fe4df0912e0f"}
Nov 03 19:06:30.055593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.055569    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:30.069899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.069876    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\") pod \"f4960f29-fa8c-4514-80ff-9f927d04ea94\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") "
Nov 03 19:06:30.070012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.069960    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\") pod \"f4960f29-fa8c-4514-80ff-9f927d04ea94\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") "
Nov 03 19:06:30.070228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.070210    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\" (UniqueName: \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\") pod \"f4960f29-fa8c-4514-80ff-9f927d04ea94\" (UID: \"f4960f29-fa8c-4514-80ff-9f927d04ea94\") "
Nov 03 19:06:30.070270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.070238    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca" pod "f4960f29-fa8c-4514-80ff-9f927d04ea94" (UID: "f4960f29-fa8c-4514-80ff-9f927d04ea94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:06:30.070828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.070784    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:30.072024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.071997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca" pod "f4960f29-fa8c-4514-80ff-9f927d04ea94" (UID: "f4960f29-fa8c-4514-80ff-9f927d04ea94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:06:30.072162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.072137    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw" pod "f4960f29-fa8c-4514-80ff-9f927d04ea94" (UID: "f4960f29-fa8c-4514-80ff-9f927d04ea94"). InnerVolumeSpecName "kube-api-access-8srgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:30.172081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.172001    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srgw\" (UniqueName: \"kubernetes.io/projected/f4960f29-fa8c-4514-80ff-9f927d04ea94-kube-api-access-8srgw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:30.172081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.172044    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4960f29-fa8c-4514-80ff-9f927d04ea94-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:30.918557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.918525    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"
Nov 03 19:06:30.918722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.918564    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj" event={"ID":"f4960f29-fa8c-4514-80ff-9f927d04ea94","Type":"ContainerDied","Data":"1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997"}
Nov 03 19:06:30.918722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:30.918655    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1590b9ccac7b5903d4bcfa082cfbcf23a30da0028285fd7331308576746aa997"
Nov 03 19:06:32.564683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:32.564639    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:35.225700 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:06:35.225661    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:06:37.212355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:37.212325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:40.048582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.048538    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"]
Nov 03 19:06:40.049462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.049438    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f4960f29-fa8c-4514-80ff-9f927d04ea94" containerName="revoke-keys"
Nov 03 19:06:40.049564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.049463    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4960f29-fa8c-4514-80ff-9f927d04ea94" containerName="revoke-keys"
Nov 03 19:06:40.049675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.049660    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4960f29-fa8c-4514-80ff-9f927d04ea94" containerName="revoke-keys"
Nov 03 19:06:40.053341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.053318    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.102263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.102224    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.102436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.102281    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7jm\" (UniqueName: \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.102436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.102377    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.108469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.108432    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"]
Nov 03 19:06:40.147407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.147371    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:06:40.147580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.147419    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:06:40.147580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.147437    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:06:40.147718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.147701    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:06:40.203526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.203486    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.203711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.203535    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7jm\" (UniqueName: \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.203711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.203630    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.204295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.204271    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.206069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.206044    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.261059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.261030    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7jm\" (UniqueName: \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\") pod \"maas-api-revoke-sub-bbbe146a-bhxzv\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.369718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.369508    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:40.747192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.747162    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"]
Nov 03 19:06:40.750826 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:40.750781    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac41f8a9_d959_4fc4_8328_2e093eae4bf6.slice/crio-ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2 WatchSource:0}: Error finding container ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2: Status 404 returned error can't find the container with id ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2
Nov 03 19:06:40.822377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:40.822340    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:41.006239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:41.006103    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv" event={"ID":"ac41f8a9-d959-4fc4-8328-2e093eae4bf6","Type":"ContainerStarted","Data":"ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2"}
Nov 03 19:06:41.904676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:41.904612    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:42.014138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:42.014102    4944 generic.go:356] "Generic (PLEG): container finished" podID="ac41f8a9-d959-4fc4-8328-2e093eae4bf6" containerID="a4b59831dc22671e90b6f9b285c1f51b5f74e3a0a495deeb0876fe3d5107b158" exitCode=0
Nov 03 19:06:42.014330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:42.014185    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv" event={"ID":"ac41f8a9-d959-4fc4-8328-2e093eae4bf6","Type":"ContainerDied","Data":"a4b59831dc22671e90b6f9b285c1f51b5f74e3a0a495deeb0876fe3d5107b158"}
Nov 03 19:06:42.049393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:42.049351    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"]
Nov 03 19:06:42.053505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:42.053478    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3860d506-q774f"]
Nov 03 19:06:43.184266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.184240    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:43.248726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.248697    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\") pod \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") "
Nov 03 19:06:43.248932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.248917    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\" (UniqueName: \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\") pod \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") "
Nov 03 19:06:43.249009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.248998    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\") pod \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\" (UID: \"ac41f8a9-d959-4fc4-8328-2e093eae4bf6\") "
Nov 03 19:06:43.249089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.249054    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca" pod "ac41f8a9-d959-4fc4-8328-2e093eae4bf6" (UID: "ac41f8a9-d959-4fc4-8328-2e093eae4bf6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:06:43.249346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.249332    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:43.251091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.251070    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca" pod "ac41f8a9-d959-4fc4-8328-2e093eae4bf6" (UID: "ac41f8a9-d959-4fc4-8328-2e093eae4bf6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:06:43.251163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.251101    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm" pod "ac41f8a9-d959-4fc4-8328-2e093eae4bf6" (UID: "ac41f8a9-d959-4fc4-8328-2e093eae4bf6"). InnerVolumeSpecName "kube-api-access-gb7jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:43.350354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.350278    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7jm\" (UniqueName: \"kubernetes.io/projected/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-kube-api-access-gb7jm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:43.350354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.350305    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac41f8a9-d959-4fc4-8328-2e093eae4bf6-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:43.868883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.868847    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:06:43.869472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.869458    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ac41f8a9-d959-4fc4-8328-2e093eae4bf6" containerName="revoke-keys"
Nov 03 19:06:43.869518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.869474    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac41f8a9-d959-4fc4-8328-2e093eae4bf6" containerName="revoke-keys"
Nov 03 19:06:43.869600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.869591    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac41f8a9-d959-4fc4-8328-2e093eae4bf6" containerName="revoke-keys"
Nov 03 19:06:43.873263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.873247    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.935860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.935830    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f" path="/var/lib/kubelet/pods/ab33c8a7-d830-4e2c-a4ae-470e4a78ae2f/volumes"
Nov 03 19:06:43.958282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958248    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958300    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958384    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958540    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958568    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8x8\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958702    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958744    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958764    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:43.958957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:43.958892    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.031310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.031264    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv" event={"ID":"ac41f8a9-d959-4fc4-8328-2e093eae4bf6","Type":"ContainerDied","Data":"ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2"}
Nov 03 19:06:44.031476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.031359    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ade3ddd788496c1d0dd37ad55c5b2691f67dd1a1bd6358c48c806823719e2ef2"
Nov 03 19:06:44.031476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.031286    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"
Nov 03 19:06:44.060557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060522    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.060711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060590    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.060711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060612    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.060711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060679    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.060897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060729    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.060897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060784    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.060908    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061020    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061048    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061061    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8x8\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061099    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061160    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061485    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.061744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.061725    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.063283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.063262    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.063371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.063297    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.074571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.074538    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:06:44.118057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.118022    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.118198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.118123    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8x8\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\") pod \"e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") " pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.187585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.187499    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:44.620601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.620567    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:06:44.625384 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:44.625353    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326443af_af71_4f81_979e_72503ca4d9b3.slice/crio-a7b91f08155d4d811f16891590292947b62620d11fff066e4c1ded1378c79cee WatchSource:0}: Error finding container a7b91f08155d4d811f16891590292947b62620d11fff066e4c1ded1378c79cee: Status 404 returned error can't find the container with id a7b91f08155d4d811f16891590292947b62620d11fff066e4c1ded1378c79cee
Nov 03 19:06:44.628890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.628851    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:06:44.628973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.628940    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:06:44.629009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:44.628980    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:06:45.040456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.040424    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" event={"ID":"326443af-af71-4f81-979e-72503ca4d9b3","Type":"ContainerStarted","Data":"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"}
Nov 03 19:06:45.040641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.040476    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" event={"ID":"326443af-af71-4f81-979e-72503ca4d9b3","Type":"ContainerStarted","Data":"a7b91f08155d4d811f16891590292947b62620d11fff066e4c1ded1378c79cee"}
Nov 03 19:06:45.187953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.187922    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:45.189336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.189306    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:45.189453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.189362    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:45.203089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:45.203034    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podStartSLOduration=2.203018346 podStartE2EDuration="2.203018346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:45.198483236 +0000 UTC m=+1303.475836820" watchObservedRunningTime="2026-11-03 19:06:45.203018346 +0000 UTC m=+1303.480371935"
Nov 03 19:06:46.152267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.152234    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"]
Nov 03 19:06:46.156950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.156927    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.188819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.188746    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:46.189176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.188841    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:46.236572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.236523    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"]
Nov 03 19:06:46.250989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.250877    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:06:46.250989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.250922    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:06:46.250989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.250950    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:06:46.251964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.251931    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:06:46.292500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.292467    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.292642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.292527    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.292718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.292697    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78j5s\" (UniqueName: \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.393928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.393897    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78j5s\" (UniqueName: \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.394109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.394019    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.394109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.394079    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.394659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.394631    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.396426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.396400    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.451171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.451101    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78j5s\" (UniqueName: \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\") pod \"maas-api-revoke-sub-a4a55ccc-h86vm\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.471724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.471697    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:46.645474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.645383    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:46.874194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:46.874162    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"]
Nov 03 19:06:46.878374 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:46.878348    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72094ff_35cd_4ed4_b40a_5a50dbc79c26.slice/crio-6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f WatchSource:0}: Error finding container 6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f: Status 404 returned error can't find the container with id 6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f
Nov 03 19:06:47.068565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:47.068521    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm" event={"ID":"e72094ff-35cd-4ed4-b40a-5a50dbc79c26","Type":"ContainerStarted","Data":"6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f"}
Nov 03 19:06:47.187969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:47.187930    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:47.188102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:47.187983    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:48.077607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:48.077571    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm" event={"ID":"e72094ff-35cd-4ed4-b40a-5a50dbc79c26","Type":"ContainerStarted","Data":"f6fc4fe678725207445867fa113edf5c838454b53cac5a9353a8e1186a5cdbb0"}
Nov 03 19:06:48.187990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:48.187957    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:48.188139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:48.188008    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:48.193602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:48.193555    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm" podStartSLOduration=2.193542704 podStartE2EDuration="2.193542704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:48.192496585 +0000 UTC m=+1306.469850171" watchObservedRunningTime="2026-11-03 19:06:48.193542704 +0000 UTC m=+1306.470896313"
Nov 03 19:06:49.087371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:49.087334    4944 generic.go:356] "Generic (PLEG): container finished" podID="e72094ff-35cd-4ed4-b40a-5a50dbc79c26" containerID="f6fc4fe678725207445867fa113edf5c838454b53cac5a9353a8e1186a5cdbb0" exitCode=0
Nov 03 19:06:49.087823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:49.087417    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm" event={"ID":"e72094ff-35cd-4ed4-b40a-5a50dbc79c26","Type":"ContainerDied","Data":"f6fc4fe678725207445867fa113edf5c838454b53cac5a9353a8e1186a5cdbb0"}
Nov 03 19:06:49.188569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:49.188536    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:49.188739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:49.188592    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:50.188401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.187964    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:50.188401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.188024    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:50.306323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.306297    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:50.486305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.486223    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\" (UniqueName: \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\") pod \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") "
Nov 03 19:06:50.486305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.486265    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\") pod \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") "
Nov 03 19:06:50.486305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.486292    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\") pod \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\" (UID: \"e72094ff-35cd-4ed4-b40a-5a50dbc79c26\") "
Nov 03 19:06:50.486751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.486711    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca" pod "e72094ff-35cd-4ed4-b40a-5a50dbc79c26" (UID: "e72094ff-35cd-4ed4-b40a-5a50dbc79c26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:06:50.487097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.487075    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:50.488431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.488403    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca" pod "e72094ff-35cd-4ed4-b40a-5a50dbc79c26" (UID: "e72094ff-35cd-4ed4-b40a-5a50dbc79c26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:06:50.488532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.488431    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s" pod "e72094ff-35cd-4ed4-b40a-5a50dbc79c26" (UID: "e72094ff-35cd-4ed4-b40a-5a50dbc79c26"). InnerVolumeSpecName "kube-api-access-78j5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:06:50.588236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.588204    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78j5s\" (UniqueName: \"kubernetes.io/projected/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-kube-api-access-78j5s\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:50.588236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:50.588231    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72094ff-35cd-4ed4-b40a-5a50dbc79c26-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:06:51.102761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.102717    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm" event={"ID":"e72094ff-35cd-4ed4-b40a-5a50dbc79c26","Type":"ContainerDied","Data":"6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f"}
Nov 03 19:06:51.102951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.102772    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd1a1b351abe9834471c6eedaef525225824bb01e2ac73422a6b5227731643f"
Nov 03 19:06:51.102951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.102735    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"
Nov 03 19:06:51.189186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.189152    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Nov 03 19:06:51.189526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.189211    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Nov 03 19:06:51.409272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:51.409190    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:52.192651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:52.192618    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:52.193132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:52.193115    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:52.193905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:52.193887    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:06:56.100673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.100631    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:06:56.866238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.866204    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"]
Nov 03 19:06:56.866920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.866903    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e72094ff-35cd-4ed4-b40a-5a50dbc79c26" containerName="revoke-keys"
Nov 03 19:06:56.866920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.866920    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72094ff-35cd-4ed4-b40a-5a50dbc79c26" containerName="revoke-keys"
Nov 03 19:06:56.867057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.867048    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72094ff-35cd-4ed4-b40a-5a50dbc79c26" containerName="revoke-keys"
Nov 03 19:06:56.870427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.870411    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:56.919699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.919663    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:06:56.919699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.919690    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:06:56.919956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.919671    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:06:56.924129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.924103    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"]
Nov 03 19:06:56.964436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.964409    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:06:56.974135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.974105    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjsr\" (UniqueName: \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:56.974244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.974172    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:56.974244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:56.974217    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.047960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.047926    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"]
Nov 03 19:06:57.051949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.051925    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83b2cdfa-2wsqr"]
Nov 03 19:06:57.076173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.076139    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.076450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.076421    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjsr\" (UniqueName: \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.076601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.076494    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.076969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.076950    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.078896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.078875    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.133781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.133710    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjsr\" (UniqueName: \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\") pod \"maas-api-revoke-sub-6c7627b8-xw72l\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.185097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.185062    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:06:57.564186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.564154    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"]
Nov 03 19:06:57.568152 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:06:57.568123    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2794aa86_81a6_4339_a806_1f2ada4d9e02.slice/crio-897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d WatchSource:0}: Error finding container 897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d: Status 404 returned error can't find the container with id 897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d
Nov 03 19:06:57.935181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:57.935152    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48a47d9-4c6d-4b40-85d6-8c769c952843" path="/var/lib/kubelet/pods/b48a47d9-4c6d-4b40-85d6-8c769c952843/volumes"
Nov 03 19:06:58.160088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:58.160004    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l" event={"ID":"2794aa86-81a6-4339-a806-1f2ada4d9e02","Type":"ContainerStarted","Data":"78cadb4705b1363fde34513d18501f8d093a9ed356e4f599a2ad12ccadcdee2b"}
Nov 03 19:06:58.160088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:58.160057    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l" event={"ID":"2794aa86-81a6-4339-a806-1f2ada4d9e02","Type":"ContainerStarted","Data":"897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d"}
Nov 03 19:06:58.268085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:58.268026    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l" podStartSLOduration=2.268012549 podStartE2EDuration="2.268012549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:06:58.267246066 +0000 UTC m=+1316.544599649" watchObservedRunningTime="2026-11-03 19:06:58.268012549 +0000 UTC m=+1316.545366132"
Nov 03 19:06:59.169323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:59.169292    4944 generic.go:356] "Generic (PLEG): container finished" podID="2794aa86-81a6-4339-a806-1f2ada4d9e02" containerID="78cadb4705b1363fde34513d18501f8d093a9ed356e4f599a2ad12ccadcdee2b" exitCode=0
Nov 03 19:06:59.169726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:06:59.169376    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l" event={"ID":"2794aa86-81a6-4339-a806-1f2ada4d9e02","Type":"ContainerDied","Data":"78cadb4705b1363fde34513d18501f8d093a9ed356e4f599a2ad12ccadcdee2b"}
Nov 03 19:07:00.327798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.327763    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:07:00.423181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.423152    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\") pod \"2794aa86-81a6-4339-a806-1f2ada4d9e02\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") "
Nov 03 19:07:00.423339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.423327    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\") pod \"2794aa86-81a6-4339-a806-1f2ada4d9e02\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") "
Nov 03 19:07:00.423396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.423381    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\" (UniqueName: \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\") pod \"2794aa86-81a6-4339-a806-1f2ada4d9e02\" (UID: \"2794aa86-81a6-4339-a806-1f2ada4d9e02\") "
Nov 03 19:07:00.423545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.423508    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca" pod "2794aa86-81a6-4339-a806-1f2ada4d9e02" (UID: "2794aa86-81a6-4339-a806-1f2ada4d9e02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:00.423860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.423842    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:00.425360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.425337    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca" pod "2794aa86-81a6-4339-a806-1f2ada4d9e02" (UID: "2794aa86-81a6-4339-a806-1f2ada4d9e02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:00.425526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.425510    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr" pod "2794aa86-81a6-4339-a806-1f2ada4d9e02" (UID: "2794aa86-81a6-4339-a806-1f2ada4d9e02"). InnerVolumeSpecName "kube-api-access-sbjsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:00.524929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.524897    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2794aa86-81a6-4339-a806-1f2ada4d9e02-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:00.524929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.524924    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbjsr\" (UniqueName: \"kubernetes.io/projected/2794aa86-81a6-4339-a806-1f2ada4d9e02-kube-api-access-sbjsr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:00.615004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:00.614960    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:01.185020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:01.184990    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"
Nov 03 19:07:01.185020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:01.184999    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l" event={"ID":"2794aa86-81a6-4339-a806-1f2ada4d9e02","Type":"ContainerDied","Data":"897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d"}
Nov 03 19:07:01.185238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:01.185086    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="897c6c86d0245419193ecbcddf2d6bfc9d23917d8e855eab3c4128a273a0b92d"
Nov 03 19:07:05.251455 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:07:05.251395    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:07:07.245942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.245768    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:07.405375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.405342    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"]
Nov 03 19:07:07.406004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.405989    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2794aa86-81a6-4339-a806-1f2ada4d9e02" containerName="revoke-keys"
Nov 03 19:07:07.406004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.406004    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2794aa86-81a6-4339-a806-1f2ada4d9e02" containerName="revoke-keys"
Nov 03 19:07:07.406121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.406112    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2794aa86-81a6-4339-a806-1f2ada4d9e02" containerName="revoke-keys"
Nov 03 19:07:07.411011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.410993    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.469887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.469851    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.470178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.470149    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.470306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.470286    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7wk\" (UniqueName: \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.470647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.470624    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"]
Nov 03 19:07:07.508859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.508762    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:07:07.508859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.508785    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:07:07.508859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.508784    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:07:07.508859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.508838    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:07:07.575515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.575474    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.576955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.576910    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.577131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.577091    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7wk\" (UniqueName: \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.577222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.577197    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.580255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.580231    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.635347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.635310    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7wk\" (UniqueName: \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\") pod \"maas-api-revoke-sub-968e48c4-7xnqm\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:07.725050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:07.725017    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:08.101617 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:08.101535    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"]
Nov 03 19:07:08.105623 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:08.105597    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c67d676_185c_48e2_92c6_467e83964e5d.slice/crio-b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6 WatchSource:0}: Error finding container b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6: Status 404 returned error can't find the container with id b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6
Nov 03 19:07:08.270566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:08.270535    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm" event={"ID":"2c67d676-185c-48e2-92c6-467e83964e5d","Type":"ContainerStarted","Data":"b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6"}
Nov 03 19:07:09.281051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:09.281013    4944 generic.go:356] "Generic (PLEG): container finished" podID="2c67d676-185c-48e2-92c6-467e83964e5d" containerID="5ea69b4a5f454614430e8d325d640556b39d91043a9b4ed6ce2e20f3a7f3d9e9" exitCode=0
Nov 03 19:07:09.281417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:09.281093    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm" event={"ID":"2c67d676-185c-48e2-92c6-467e83964e5d","Type":"ContainerDied","Data":"5ea69b4a5f454614430e8d325d640556b39d91043a9b4ed6ce2e20f3a7f3d9e9"}
Nov 03 19:07:10.468881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.468855    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:10.626940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.626866    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\") pod \"2c67d676-185c-48e2-92c6-467e83964e5d\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") "
Nov 03 19:07:10.627069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.626979    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\" (UniqueName: \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\") pod \"2c67d676-185c-48e2-92c6-467e83964e5d\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") "
Nov 03 19:07:10.627104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.627079    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\") pod \"2c67d676-185c-48e2-92c6-467e83964e5d\" (UID: \"2c67d676-185c-48e2-92c6-467e83964e5d\") "
Nov 03 19:07:10.627219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.627198    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca" pod "2c67d676-185c-48e2-92c6-467e83964e5d" (UID: "2c67d676-185c-48e2-92c6-467e83964e5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:10.627526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.627507    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:10.629130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.629108    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk" pod "2c67d676-185c-48e2-92c6-467e83964e5d" (UID: "2c67d676-185c-48e2-92c6-467e83964e5d"). InnerVolumeSpecName "kube-api-access-wn7wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:10.629239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.629223    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca" pod "2c67d676-185c-48e2-92c6-467e83964e5d" (UID: "2c67d676-185c-48e2-92c6-467e83964e5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:10.728669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.728638    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7wk\" (UniqueName: \"kubernetes.io/projected/2c67d676-185c-48e2-92c6-467e83964e5d-kube-api-access-wn7wk\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:10.728669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:10.728669    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c67d676-185c-48e2-92c6-467e83964e5d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:11.297070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:11.297029    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm" event={"ID":"2c67d676-185c-48e2-92c6-467e83964e5d","Type":"ContainerDied","Data":"b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6"}
Nov 03 19:07:11.297070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:11.297049    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"
Nov 03 19:07:11.297431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:11.297148    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b420bbc5acd8af8ea6b3c2776b5cc4f4aa136939013a937a12d483830305dbf6"
Nov 03 19:07:11.847510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:11.847479    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:12.053614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:12.053577    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"]
Nov 03 19:07:12.057701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:12.057678    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cc8bf0a-7sfdj"]
Nov 03 19:07:13.935709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:13.935673    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323d736a-94c4-4105-9cfa-bbafdb54dcd4" path="/var/lib/kubelet/pods/323d736a-94c4-4105-9cfa-bbafdb54dcd4/volumes"
Nov 03 19:07:15.049584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.049548    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"]
Nov 03 19:07:15.095598 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.095566    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372cc8ad-n722r"]
Nov 03 19:07:15.473785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.473704    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"]
Nov 03 19:07:15.474420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.474402    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2c67d676-185c-48e2-92c6-467e83964e5d" containerName="revoke-keys"
Nov 03 19:07:15.474465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.474421    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c67d676-185c-48e2-92c6-467e83964e5d" containerName="revoke-keys"
Nov 03 19:07:15.474560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.474551    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c67d676-185c-48e2-92c6-467e83964e5d" containerName="revoke-keys"
Nov 03 19:07:15.478955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.478937    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.489483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.489447    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8chqq\" (UniqueName: \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.489595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.489576    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.489659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.489643    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.529120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.529093    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:07:15.529264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.529149    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:07:15.529264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.529161    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:07:15.529264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.529161    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:07:15.532014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.531993    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"]
Nov 03 19:07:15.591153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.591127    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8chqq\" (UniqueName: \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.591307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.591198    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.591307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.591245    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.591930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.591908    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.593529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.593507    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.648842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.648783    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8chqq\" (UniqueName: \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\") pod \"maas-api-revoke-sub-fb26413f-nzfzs\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.793043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.793014    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:15.936025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:15.935993    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d772ec7e-cf47-4717-a651-a767dd391911" path="/var/lib/kubelet/pods/d772ec7e-cf47-4717-a651-a767dd391911/volumes"
Nov 03 19:07:16.171774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:16.171700    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"]
Nov 03 19:07:16.175580 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:16.175552    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c7cd95_a084_4614_a208_7ca4965cba2b.slice/crio-c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4 WatchSource:0}: Error finding container c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4: Status 404 returned error can't find the container with id c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4
Nov 03 19:07:16.335146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:16.335103    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs" event={"ID":"c9c7cd95-a084-4614-a208-7ca4965cba2b","Type":"ContainerStarted","Data":"c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4"}
Nov 03 19:07:16.467755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:16.467722    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:17.344505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:17.344468    4944 generic.go:356] "Generic (PLEG): container finished" podID="c9c7cd95-a084-4614-a208-7ca4965cba2b" containerID="c9cb07c27131ca38b30322138b856ad2c34f2a24ebe2f0be6d74d029d4431aab" exitCode=0
Nov 03 19:07:17.344944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:17.344553    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs" event={"ID":"c9c7cd95-a084-4614-a208-7ca4965cba2b","Type":"ContainerDied","Data":"c9cb07c27131ca38b30322138b856ad2c34f2a24ebe2f0be6d74d029d4431aab"}
Nov 03 19:07:18.514265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.514240    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:18.535700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.535672    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\" (UniqueName: \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\") pod \"c9c7cd95-a084-4614-a208-7ca4965cba2b\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") "
Nov 03 19:07:18.535850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.535734    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\") pod \"c9c7cd95-a084-4614-a208-7ca4965cba2b\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") "
Nov 03 19:07:18.535913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.535896    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\") pod \"c9c7cd95-a084-4614-a208-7ca4965cba2b\" (UID: \"c9c7cd95-a084-4614-a208-7ca4965cba2b\") "
Nov 03 19:07:18.536203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.536182    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca" pod "c9c7cd95-a084-4614-a208-7ca4965cba2b" (UID: "c9c7cd95-a084-4614-a208-7ca4965cba2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:18.536537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.536517    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:18.537724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.537700    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca" pod "c9c7cd95-a084-4614-a208-7ca4965cba2b" (UID: "c9c7cd95-a084-4614-a208-7ca4965cba2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:18.537825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.537708    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq" pod "c9c7cd95-a084-4614-a208-7ca4965cba2b" (UID: "c9c7cd95-a084-4614-a208-7ca4965cba2b"). InnerVolumeSpecName "kube-api-access-8chqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:18.637948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.637872    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8chqq\" (UniqueName: \"kubernetes.io/projected/c9c7cd95-a084-4614-a208-7ca4965cba2b-kube-api-access-8chqq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:18.637948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:18.637901    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c7cd95-a084-4614-a208-7ca4965cba2b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:19.360816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:19.360774    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"
Nov 03 19:07:19.360816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:19.360773    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs" event={"ID":"c9c7cd95-a084-4614-a208-7ca4965cba2b","Type":"ContainerDied","Data":"c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4"}
Nov 03 19:07:19.361061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:19.360847    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c78a3d83c155ccb1313d0313e239476f1519921151014584f1b67e84ab34f5b4"
Nov 03 19:07:20.503856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.503817    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:07:20.504460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.504445    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c9c7cd95-a084-4614-a208-7ca4965cba2b" containerName="revoke-keys"
Nov 03 19:07:20.504504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.504461    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c7cd95-a084-4614-a208-7ca4965cba2b" containerName="revoke-keys"
Nov 03 19:07:20.504571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.504563    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c7cd95-a084-4614-a208-7ca4965cba2b" containerName="revoke-keys"
Nov 03 19:07:20.508233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.508217    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.557216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.557174    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"openshift-service-ca.crt\""
Nov 03 19:07:20.562782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.562749    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.562940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.562819    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.562940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.562852    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.562940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.562906    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.563060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.562959    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57scn\" (UniqueName: \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.563060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.563006    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.603089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.603060    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"kube-root-ca.crt\""
Nov 03 19:07:20.603271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.603227    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"default-dockercfg-fbmhw\""
Nov 03 19:07:20.603271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.603227    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-90a9bf\"/\"distinct-2-w5-90a9bf-kserve-self-signed-certs\""
Nov 03 19:07:20.607965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.607946    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:07:20.665824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.665768    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57scn\" (UniqueName: \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.666150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.666115    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.666602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.666563    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.666822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.666804    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.666964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.666947    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.667183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.667168    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.667778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.667757    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.669217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.669196    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.669689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.669637    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.676738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.676711    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.678988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.678958    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.703849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.703778    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:20.725199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.725172    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57scn\" (UniqueName: \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\") pod \"distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") " pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.820797 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.820692    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:20.996438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:20.996401    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:21.205058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:21.204979    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:07:21.208702 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:21.208675    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8e7c020_472d_4131_806d_8ea0847e3a3d.slice/crio-0ae006a3e9cde63fd33d5e954b7e7aa971648f060ec96a6b1c144d0ba9d11260 WatchSource:0}: Error finding container 0ae006a3e9cde63fd33d5e954b7e7aa971648f060ec96a6b1c144d0ba9d11260: Status 404 returned error can't find the container with id 0ae006a3e9cde63fd33d5e954b7e7aa971648f060ec96a6b1c144d0ba9d11260
Nov 03 19:07:21.383070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:21.383034    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" event={"ID":"a8e7c020-472d-4131-806d-8ea0847e3a3d","Type":"ContainerStarted","Data":"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"}
Nov 03 19:07:21.383238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:21.383083    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" event={"ID":"a8e7c020-472d-4131-806d-8ea0847e3a3d","Type":"ContainerStarted","Data":"0ae006a3e9cde63fd33d5e954b7e7aa971648f060ec96a6b1c144d0ba9d11260"}
Nov 03 19:07:21.514537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:21.514488    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podStartSLOduration=1.514475145 podStartE2EDuration="1.514475145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:21.512123886 +0000 UTC m=+1339.789477504" watchObservedRunningTime="2026-11-03 19:07:21.514475145 +0000 UTC m=+1339.791828727"
Nov 03 19:07:21.821017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:21.820937    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:23.042655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.042624    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"]
Nov 03 19:07:23.047047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.047030    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.102414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.102382    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"]
Nov 03 19:07:23.124134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.124101    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtrz\" (UniqueName: \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.124304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.124286    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.124380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.124332    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.141724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.141690    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:07:23.141893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.141782    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:07:23.141893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.141850    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:07:23.141893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.141883    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:07:23.225760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.225729    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phtrz\" (UniqueName: \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.225926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.225865    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.225926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.225896    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.226509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.226482    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.228308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.228281    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.278914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.278885    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:23.282694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.282672    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtrz\" (UniqueName: \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\") pod \"maas-api-revoke-sub-2ae8152d-5579q\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.361766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.361704    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:23.524145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.524112    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:23.741695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:23.741666    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"]
Nov 03 19:07:23.745983 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:23.745950    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2544645_038a_4c1d_9ce2_3b2bdd1624a1.slice/crio-093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a WatchSource:0}: Error finding container 093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a: Status 404 returned error can't find the container with id 093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a
Nov 03 19:07:24.412718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:24.412630    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q" event={"ID":"b2544645-038a-4c1d-9ce2-3b2bdd1624a1","Type":"ContainerStarted","Data":"830a8ab7676c5e1123652e897a9757fd37f77f4b7e9d6830622ae530fbf6d194"}
Nov 03 19:07:24.412718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:24.412688    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q" event={"ID":"b2544645-038a-4c1d-9ce2-3b2bdd1624a1","Type":"ContainerStarted","Data":"093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a"}
Nov 03 19:07:24.521102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:24.521053    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q" podStartSLOduration=1.521035834 podStartE2EDuration="1.521035834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:24.520496664 +0000 UTC m=+1342.797850246" watchObservedRunningTime="2026-11-03 19:07:24.521035834 +0000 UTC m=+1342.798389416"
Nov 03 19:07:25.420945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:25.420913    4944 generic.go:356] "Generic (PLEG): container finished" podID="b2544645-038a-4c1d-9ce2-3b2bdd1624a1" containerID="830a8ab7676c5e1123652e897a9757fd37f77f4b7e9d6830622ae530fbf6d194" exitCode=0
Nov 03 19:07:25.421336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:25.421003    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q" event={"ID":"b2544645-038a-4c1d-9ce2-3b2bdd1624a1","Type":"ContainerDied","Data":"830a8ab7676c5e1123652e897a9757fd37f77f4b7e9d6830622ae530fbf6d194"}
Nov 03 19:07:25.664221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:25.664175    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:26.047339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.047304    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"]
Nov 03 19:07:26.093298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.093262    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3737e1b9-d7hkl"]
Nov 03 19:07:26.590672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.590646    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:26.671819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.671771    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\") pod \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") "
Nov 03 19:07:26.671980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.671962    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\" (UniqueName: \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\") pod \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") "
Nov 03 19:07:26.672136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.672120    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\") pod \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\" (UID: \"b2544645-038a-4c1d-9ce2-3b2bdd1624a1\") "
Nov 03 19:07:26.672488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.672463    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca" pod "b2544645-038a-4c1d-9ce2-3b2bdd1624a1" (UID: "b2544645-038a-4c1d-9ce2-3b2bdd1624a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:26.672740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.672723    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:26.673912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.673894    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca" pod "b2544645-038a-4c1d-9ce2-3b2bdd1624a1" (UID: "b2544645-038a-4c1d-9ce2-3b2bdd1624a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:26.673979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.673917    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz" pod "b2544645-038a-4c1d-9ce2-3b2bdd1624a1" (UID: "b2544645-038a-4c1d-9ce2-3b2bdd1624a1"). InnerVolumeSpecName "kube-api-access-phtrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:26.774410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.774380    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:26.774410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:26.774408    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phtrz\" (UniqueName: \"kubernetes.io/projected/b2544645-038a-4c1d-9ce2-3b2bdd1624a1-kube-api-access-phtrz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:27.437558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:27.437529    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"
Nov 03 19:07:27.437723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:27.437564    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q" event={"ID":"b2544645-038a-4c1d-9ce2-3b2bdd1624a1","Type":"ContainerDied","Data":"093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a"}
Nov 03 19:07:27.437723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:27.437613    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="093638c31d2186e77d815663a54dfac6caf9d28d4ab92a7b159c58af4ca87e9a"
Nov 03 19:07:27.935917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:27.935878    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8e07d8-cf94-4d5f-9bfc-8254266629cb" path="/var/lib/kubelet/pods/7d8e07d8-cf94-4d5f-9bfc-8254266629cb/volumes"
Nov 03 19:07:28.831104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.831068    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:07:28.831765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.831749    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b2544645-038a-4c1d-9ce2-3b2bdd1624a1" containerName="revoke-keys"
Nov 03 19:07:28.831846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.831767    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2544645-038a-4c1d-9ce2-3b2bdd1624a1" containerName="revoke-keys"
Nov 03 19:07:28.831884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.831875    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2544645-038a-4c1d-9ce2-3b2bdd1624a1" containerName="revoke-keys"
Nov 03 19:07:28.835841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.835822    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:28.892860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:28.892829    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:07:29.009187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009141    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009189    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009214    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009301    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009330    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009358    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009482    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqv6\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009655    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.009758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.009716    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.111857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.111755    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.111889    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.111972    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.111997    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112012    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112050    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112100    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112146    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112204    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqv6\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112207    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112288    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112414    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112575    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.112709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.112686    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.114163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.114139    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.114623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.114606    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.169137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.169105    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.169247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.169220    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqv6\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\") pod \"e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") " pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.390847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.390745    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"]
Nov 03 19:07:29.396497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.396475    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.448038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.448008    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:07:29.448038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.448007    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:07:29.448258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.448032    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:07:29.448258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.448114    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:07:29.448432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.448413    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"]
Nov 03 19:07:29.449529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.449505    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:29.519072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.519036    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrsd\" (UniqueName: \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.524925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.519092    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.524925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.519231    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.620605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.620577    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrsd\" (UniqueName: \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.620771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.620614    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.620771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.620658    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.621349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.621321    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.623026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.622996    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.677399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.677332    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrsd\" (UniqueName: \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\") pod \"maas-api-revoke-sub-6c9e7f23-cfrqh\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.712511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.712486    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:29.881652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.881623    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:07:29.885087 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:29.885057    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d993953_d8bc_4c7d_bee4_fd269048ce7f.slice/crio-cd639cde269e37bb7e1d21546265407dd07228720bd5fdd865c182cefcf9a93d WatchSource:0}: Error finding container cd639cde269e37bb7e1d21546265407dd07228720bd5fdd865c182cefcf9a93d: Status 404 returned error can't find the container with id cd639cde269e37bb7e1d21546265407dd07228720bd5fdd865c182cefcf9a93d
Nov 03 19:07:29.886039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.886015    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:07:29.887778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.887749    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:29.887862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.887848    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:29.887910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.887878    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:29.891954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.891938    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:29.914046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.914024    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:07:29.918903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.918886    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:29.984466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:29.984435    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.029851    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.029953    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbckt\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.029996    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030042    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030069    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030129    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030177    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030204    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030253    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030329    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030361    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030391    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030647    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030690    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030703    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030737    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.030837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030763    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.031958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030895    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.031958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.030977    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztr89\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.086752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.086570    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"]
Nov 03 19:07:30.089990 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:30.089963    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4930c3_883b_4d35_b8b1_259113efe96d.slice/crio-57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7 WatchSource:0}: Error finding container 57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7: Status 404 returned error can't find the container with id 57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7
Nov 03 19:07:30.132158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132133    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132176    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132195    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132241    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132307    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132344    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132372    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132426    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132461    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132507    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132533    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132563    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132599    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztr89\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.132674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132612    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132761    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132874    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbckt\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.132937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132920    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.132973    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133000    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133238    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133515    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133530    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133555    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133781    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.133951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.133858    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.134250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.134032    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.134250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.134058    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.134421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.134397    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.134783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.134766    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.135233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.135085    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.135233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.135115    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.135569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.135553    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.190261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.190189    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.190413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.190342    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.190641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.190618    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztr89\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.191521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.191498    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbckt\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\") pod \"e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") " pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.224630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.224584    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:30.265373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.265341    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:30.268401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.268368    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:30.470030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.469936    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh" event={"ID":"ec4930c3-883b-4d35-b8b1-259113efe96d","Type":"ContainerStarted","Data":"57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7"}
Nov 03 19:07:30.472492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.472172    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" event={"ID":"3d993953-d8bc-4c7d-bee4-fd269048ce7f","Type":"ContainerStarted","Data":"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"}
Nov 03 19:07:30.472492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.472232    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" event={"ID":"3d993953-d8bc-4c7d-bee4-fd269048ce7f","Type":"ContainerStarted","Data":"cd639cde269e37bb7e1d21546265407dd07228720bd5fdd865c182cefcf9a93d"}
Nov 03 19:07:30.629646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.629579    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podStartSLOduration=2.629558309 podStartE2EDuration="2.629558309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:30.625504236 +0000 UTC m=+1348.902857819" watchObservedRunningTime="2026-11-03 19:07:30.629558309 +0000 UTC m=+1348.906911892"
Nov 03 19:07:30.748944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.748912    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:07:30.755403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.755376    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:07:30.756467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.756429    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.756520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.756485    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.756587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.756519    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.758745 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:30.758718    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc76a066_aca4_4ce2_8bfe_4f9d06cae22e.slice/crio-c81054893b4398be0ddff38867635b62f737fb9ff72df224a93c5947c2b0d361 WatchSource:0}: Error finding container c81054893b4398be0ddff38867635b62f737fb9ff72df224a93c5947c2b0d361: Status 404 returned error can't find the container with id c81054893b4398be0ddff38867635b62f737fb9ff72df224a93c5947c2b0d361
Nov 03 19:07:30.761138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.761101    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.761242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.761173    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.761242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.761213    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:07:30.821766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.821746    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:30.837119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:30.837095    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:31.450163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.450080    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:31.451970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.451945    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:31.452071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.451997    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:31.481445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.481408    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" event={"ID":"2226d542-31a5-4042-8aa7-43b49d0e5bac","Type":"ContainerStarted","Data":"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"}
Nov 03 19:07:31.481594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.481473    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" event={"ID":"2226d542-31a5-4042-8aa7-43b49d0e5bac","Type":"ContainerStarted","Data":"a7fc93946cc6460f9a6fdcc5b83151add9686acbc99b5ae27db1547d396d3411"}
Nov 03 19:07:31.483160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.483135    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" event={"ID":"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e","Type":"ContainerStarted","Data":"7b0b0f2697c4b2b453d66e23968da2f7a14120eaa9063d40dce5a0dc0b861589"}
Nov 03 19:07:31.483275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.483178    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" event={"ID":"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e","Type":"ContainerStarted","Data":"c81054893b4398be0ddff38867635b62f737fb9ff72df224a93c5947c2b0d361"}
Nov 03 19:07:31.485776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.485661    4944 generic.go:356] "Generic (PLEG): container finished" podID="ec4930c3-883b-4d35-b8b1-259113efe96d" containerID="05fde70626bd0d4e12d5eab764a19e671966e738e309db2913e8f2c872a9b90c" exitCode=0
Nov 03 19:07:31.485925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.485739    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh" event={"ID":"ec4930c3-883b-4d35-b8b1-259113efe96d","Type":"ContainerDied","Data":"05fde70626bd0d4e12d5eab764a19e671966e738e309db2913e8f2c872a9b90c"}
Nov 03 19:07:31.498150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.498128    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:31.605337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.605281    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podStartSLOduration=2.605265945 podStartE2EDuration="2.605265945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:31.599759293 +0000 UTC m=+1349.877112902" watchObservedRunningTime="2026-11-03 19:07:31.605265945 +0000 UTC m=+1349.882619527"
Nov 03 19:07:31.903775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:31.903249    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podStartSLOduration=2.903229172 podStartE2EDuration="2.903229172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:31.899392938 +0000 UTC m=+1350.176746537" watchObservedRunningTime="2026-11-03 19:07:31.903229172 +0000 UTC m=+1350.180582758"
Nov 03 19:07:32.097344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.097304    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"]
Nov 03 19:07:32.103074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.103039    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b87a4115-2mgr7"]
Nov 03 19:07:32.229872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.228987    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:32.233810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.233149    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:32.233810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.233209    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:32.284846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.281383    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:32.289922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.287120    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:32.289922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.287182    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:32.450815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.450273    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:32.450815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:32.450342    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:33.098514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.098484    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:33.204197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.204165    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\" (UniqueName: \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\") pod \"ec4930c3-883b-4d35-b8b1-259113efe96d\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") "
Nov 03 19:07:33.204403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.204220    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\") pod \"ec4930c3-883b-4d35-b8b1-259113efe96d\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") "
Nov 03 19:07:33.204403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.204328    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\") pod \"ec4930c3-883b-4d35-b8b1-259113efe96d\" (UID: \"ec4930c3-883b-4d35-b8b1-259113efe96d\") "
Nov 03 19:07:33.204774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.204748    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca" pod "ec4930c3-883b-4d35-b8b1-259113efe96d" (UID: "ec4930c3-883b-4d35-b8b1-259113efe96d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:33.206545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.206521    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd" pod "ec4930c3-883b-4d35-b8b1-259113efe96d" (UID: "ec4930c3-883b-4d35-b8b1-259113efe96d"). InnerVolumeSpecName "kube-api-access-tqrsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:33.238826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.234347    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:33.238826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.234401    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:33.239043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.238902    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca" pod "ec4930c3-883b-4d35-b8b1-259113efe96d" (UID: "ec4930c3-883b-4d35-b8b1-259113efe96d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:33.279871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.270806    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:33.280124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.280091    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:33.306370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.306333    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrsd\" (UniqueName: \"kubernetes.io/projected/ec4930c3-883b-4d35-b8b1-259113efe96d-kube-api-access-tqrsd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.306517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.306373    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.306517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.306390    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4930c3-883b-4d35-b8b1-259113efe96d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:33.459815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.459667    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:33.459815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.459727    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:33.524000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.523961    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh" event={"ID":"ec4930c3-883b-4d35-b8b1-259113efe96d","Type":"ContainerDied","Data":"57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7"}
Nov 03 19:07:33.524231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.524212    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57f17287c2b756fbbacba832c7dfb92473d1cea9ad82c86731a62cadc74097c7"
Nov 03 19:07:33.544882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.525223    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"
Nov 03 19:07:33.987858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:33.987536    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2919be-f379-4be9-b7ba-3f2e209acd13" path="/var/lib/kubelet/pods/8a2919be-f379-4be9-b7ba-3f2e209acd13/volumes"
Nov 03 19:07:34.233817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.231323    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:34.233817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.231418    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:34.278817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.278649    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:34.278817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.278710    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:34.459131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.459084    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:34.460164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:34.459441    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:35.209044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.185370    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:35.276812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.274064    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:35.276812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.274146    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:35.378820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.361531    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:35.378820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.362157    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:35.468868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.450809    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:35.468868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.451226    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:35.902949 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:07:35.902892    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:07:35.922153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:35.913275    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.106889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.106843    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.195630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.195526    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.228686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.228652    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:36.229015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.228986    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:36.281074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.281038    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:36.281323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.281292    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:36.295911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.295868    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.352811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.341058    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.352811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.348866    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.411726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.411347    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.490879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.471686    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:36.490879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.471750    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:36.514818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.513859    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.653828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.536460    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.763867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.737276    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:36.802733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.802394    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.833774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.833730    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:07:36.882154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.882110    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.925492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.925451    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.925913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.925874    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:36.926154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:36.926133    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:07:37.090872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.089930    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"]
Nov 03 19:07:37.110952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.110900    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.111641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.111609    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:07:37.204405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.204358    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.224811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.224435    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:07:37.239864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.226549    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:37.239864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.226603    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:37.287842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.276097    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:37.287842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.276152    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:37.441871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.435133    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.441871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.437688    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.452346    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.452635    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.452925    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.452984    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.455147    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:07:37.465129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.455352    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:07:37.555864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.555325    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.555864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.555700    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:07:37.556398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.555937    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:07:37.556398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.555999    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:37.683289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.654971    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caff934c-fxp9f"]
Nov 03 19:07:37.683289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.656518    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:37.683289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.656566    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:37.773873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.738318    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.841911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.841864    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.888864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.886204    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:07:37.888864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.887192    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:37.888864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.887504    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:07:37.925918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.925863    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:07:37.951825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.938047    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 03 19:07:37.974602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:37.967643    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.161984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.148180    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:38.212092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.212041    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:38.298981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.298940    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:38.325918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.325866    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:38.326174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.314087    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:38.326339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.326316    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.439061    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d97b1dac-a239-4ff0-a1a3-daa45766d41a" path="/var/lib/kubelet/pods/d97b1dac-a239-4ff0-a1a3-daa45766d41a/volumes"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.441949    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.444926    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.465427    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.466063    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.488876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.466484    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.565813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.556023    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:38.565813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.556360    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:07:38.628814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.582935    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:38.628814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.582999    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:38.853886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.853845    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.022815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.887672    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:39.022815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.010111    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:39.079473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:38.983831    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.235924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.186134    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:39.235924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.231923    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:39.235924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.231980    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:39.235924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.232089    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.278853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.274131    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:39.278853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.274191    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:39.496251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.474443    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.550194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.484039    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.617763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.484082    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:07:39.626811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.484135    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.497938    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.257799    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.562028    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.561971    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.702398    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.258403    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.256969    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:39.673571    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.118964    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.221462    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.260176    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.261816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.260404    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.507200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.507130    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.515461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.515333    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Nov 03 19:07:40.532084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.532047    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Nov 03 19:07:40.562266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.562222    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:40.562551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.562504    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:40.618292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.618242    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.619036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.619002    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:40.639195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.639153    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:07:40.640032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.640000    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": context deadline exceeded"
Nov 03 19:07:40.644952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.644925    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:40.649897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.649865    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:40.949514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.947655    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:07:40.968062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.965609    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:07:40.968062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:40.965669    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:07:41.010353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.010320    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:07:41.011586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.011562    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:07:41.114824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.104134    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:07:41.236855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.232816    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 03 19:07:41.236855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.232872    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 03 19:07:41.247818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.244783    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:07:41.251811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.250560    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:07:41.274881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.260093    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:41.274881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.260339    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:07:41.274881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.271661    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Nov 03 19:07:41.274881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.271722    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Nov 03 19:07:41.337517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.294069    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:07:41.361906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.361592    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:07:41.577033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.573746    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:07:41.593418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:41.593369    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:07:42.062152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:42.062051    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"]
Nov 03 19:07:42.071891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:42.071851    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e237ec62-xwx55"]
Nov 03 19:07:42.230011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:42.229975    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:42.232508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:42.231345    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:42.235885 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:42.235669    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:07:43.225842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:43.225802    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:07:43.226344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:43.225864    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:43.266682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:43.266644    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:07:43.266889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:43.266698    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:43.943550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:43.943498    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c3b539-f889-44cf-9dd5-e676eadc7427" path="/var/lib/kubelet/pods/10c3b539-f889-44cf-9dd5-e676eadc7427/volumes"
Nov 03 19:07:44.315020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.314977    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:07:44.315697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.314979    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:07:44.315847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.315710    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:44.316109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.315652    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:07:44.427467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.427431    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:44.429080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.429055    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:44.430259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.430233    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:07:44.497100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.497067    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:44.497403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.497382    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:44.498611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.498590    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:07:44.937513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:44.937478    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:47.050819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:47.050759    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"]
Nov 03 19:07:47.096744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:47.096699    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2b59c18-vbtf5"]
Nov 03 19:07:47.242277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:47.242229    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:47.936847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:47.936813    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164acba0-9da8-4c91-98cf-234fb7dc56eb" path="/var/lib/kubelet/pods/164acba0-9da8-4c91-98cf-234fb7dc56eb/volumes"
Nov 03 19:07:48.047065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:48.047030    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"]
Nov 03 19:07:48.093388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:48.093354    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab58f010-4hcx6"]
Nov 03 19:07:49.935292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:49.935255    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e665dc-3517-4704-9320-853689c8cf66" path="/var/lib/kubelet/pods/a7e665dc-3517-4704-9320-853689c8cf66/volumes"
Nov 03 19:07:50.045726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:50.045692    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"]
Nov 03 19:07:50.091829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:50.091769    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50399228-q5q55"]
Nov 03 19:07:51.375423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.375391    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"]
Nov 03 19:07:51.376124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.376106    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ec4930c3-883b-4d35-b8b1-259113efe96d" containerName="revoke-keys"
Nov 03 19:07:51.376169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.376126    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4930c3-883b-4d35-b8b1-259113efe96d" containerName="revoke-keys"
Nov 03 19:07:51.376271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.376262    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4930c3-883b-4d35-b8b1-259113efe96d" containerName="revoke-keys"
Nov 03 19:07:51.380035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.380019    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.437102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.437070    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"]
Nov 03 19:07:51.476876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.476841    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:07:51.477096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.476871    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:07:51.477172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.476871    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:07:51.477172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.476910    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:07:51.485674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.485652    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.485820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.485688    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.485971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.485951    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnz2\" (UniqueName: \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.587463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.587422    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.587687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.587477    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.587776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.587759    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnz2\" (UniqueName: \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.588243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.588221    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.589775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.589754    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.647745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.647664    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnz2\" (UniqueName: \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\") pod \"maas-api-revoke-sub-21944872-zbvrv\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.677067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.677035    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:51.694118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.694083    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:51.940066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:51.939982    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f8c208-2052-45e7-b0e8-34ccfcd8a502" path="/var/lib/kubelet/pods/c2f8c208-2052-45e7-b0e8-34ccfcd8a502/volumes"
Nov 03 19:07:52.085849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.085807    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"]
Nov 03 19:07:52.090241 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:52.090206    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d18983_de37_422b_9928_78ff67798770.slice/crio-0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e WatchSource:0}: Error finding container 0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e: Status 404 returned error can't find the container with id 0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e
Nov 03 19:07:52.720825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.720736    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"]
Nov 03 19:07:52.725548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.725530    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.779135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.779105    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"]
Nov 03 19:07:52.807494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.807457    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8l4d\" (UniqueName: \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.807706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.807677    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.807921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.807901    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.852528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.852493    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv" event={"ID":"c1d18983-de37-422b-9928-78ff67798770","Type":"ContainerStarted","Data":"037114c9ca2647b8c41198065232b9e690774c21be7f64e99fc5b467368b8cc0"}
Nov 03 19:07:52.852669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.852554    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv" event={"ID":"c1d18983-de37-422b-9928-78ff67798770","Type":"ContainerStarted","Data":"0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e"}
Nov 03 19:07:52.909339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.909312    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8l4d\" (UniqueName: \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.909487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.909431    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.909554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.909539    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.910250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.910225    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.911815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.911776    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:52.963674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.963612    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv" podStartSLOduration=1.963599303 podStartE2EDuration="1.963599303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:52.960178875 +0000 UTC m=+1371.237532459" watchObservedRunningTime="2026-11-03 19:07:52.963599303 +0000 UTC m=+1371.240952885"
Nov 03 19:07:52.968590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:52.968561    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8l4d\" (UniqueName: \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\") pod \"maas-api-revoke-sub-30b116d3-cwss4\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:53.040538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.040504    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:53.419040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.418966    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"]
Nov 03 19:07:53.422841 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:53.422784    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod558451a8_f2ef_435b_9699_04ebdec26e05.slice/crio-c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f WatchSource:0}: Error finding container c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f: Status 404 returned error can't find the container with id c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f
Nov 03 19:07:53.862184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.862150    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4" event={"ID":"558451a8-f2ef-435b-9699-04ebdec26e05","Type":"ContainerStarted","Data":"ecacfa93ff1d0023283a83f90186ad147b8f65165d28bb9e5e5852bab74ffc83"}
Nov 03 19:07:53.862599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.862203    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4" event={"ID":"558451a8-f2ef-435b-9699-04ebdec26e05","Type":"ContainerStarted","Data":"c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f"}
Nov 03 19:07:53.863830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.863805    4944 generic.go:356] "Generic (PLEG): container finished" podID="c1d18983-de37-422b-9928-78ff67798770" containerID="037114c9ca2647b8c41198065232b9e690774c21be7f64e99fc5b467368b8cc0" exitCode=0
Nov 03 19:07:53.863929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.863837    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv" event={"ID":"c1d18983-de37-422b-9928-78ff67798770","Type":"ContainerDied","Data":"037114c9ca2647b8c41198065232b9e690774c21be7f64e99fc5b467368b8cc0"}
Nov 03 19:07:53.973109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:53.973012    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4" podStartSLOduration=1.972998252 podStartE2EDuration="1.972998252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:07:53.971106974 +0000 UTC m=+1372.248460556" watchObservedRunningTime="2026-11-03 19:07:53.972998252 +0000 UTC m=+1372.250351835"
Nov 03 19:07:54.872421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:54.872386    4944 generic.go:356] "Generic (PLEG): container finished" podID="558451a8-f2ef-435b-9699-04ebdec26e05" containerID="ecacfa93ff1d0023283a83f90186ad147b8f65165d28bb9e5e5852bab74ffc83" exitCode=0
Nov 03 19:07:54.872827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:54.872474    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4" event={"ID":"558451a8-f2ef-435b-9699-04ebdec26e05","Type":"ContainerDied","Data":"ecacfa93ff1d0023283a83f90186ad147b8f65165d28bb9e5e5852bab74ffc83"}
Nov 03 19:07:55.036890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.036855    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:55.140839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.140722    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\") pod \"c1d18983-de37-422b-9928-78ff67798770\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") "
Nov 03 19:07:55.140839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.140797    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\" (UniqueName: \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\") pod \"c1d18983-de37-422b-9928-78ff67798770\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") "
Nov 03 19:07:55.141047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.140957    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\") pod \"c1d18983-de37-422b-9928-78ff67798770\" (UID: \"c1d18983-de37-422b-9928-78ff67798770\") "
Nov 03 19:07:55.141116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.141080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca" pod "c1d18983-de37-422b-9928-78ff67798770" (UID: "c1d18983-de37-422b-9928-78ff67798770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:55.141652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.141635    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d18983-de37-422b-9928-78ff67798770-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:55.142889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.142864    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2" pod "c1d18983-de37-422b-9928-78ff67798770" (UID: "c1d18983-de37-422b-9928-78ff67798770"). InnerVolumeSpecName "kube-api-access-2jnz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:55.143001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.142981    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca" pod "c1d18983-de37-422b-9928-78ff67798770" (UID: "c1d18983-de37-422b-9928-78ff67798770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:55.243412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.243379    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d18983-de37-422b-9928-78ff67798770-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:55.243412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.243408    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnz2\" (UniqueName: \"kubernetes.io/projected/c1d18983-de37-422b-9928-78ff67798770-kube-api-access-2jnz2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:55.881533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.881497    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"
Nov 03 19:07:55.881533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.881509    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv" event={"ID":"c1d18983-de37-422b-9928-78ff67798770","Type":"ContainerDied","Data":"0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e"}
Nov 03 19:07:55.882066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:55.881597    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8485f0319a5229bc7f60b076e53090982410293a02c9e572e8e3ddf8f84e6e"
Nov 03 19:07:56.044165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.044140    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:56.156906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.156811    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\") pod \"558451a8-f2ef-435b-9699-04ebdec26e05\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") "
Nov 03 19:07:56.157077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.156940    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\") pod \"558451a8-f2ef-435b-9699-04ebdec26e05\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") "
Nov 03 19:07:56.157239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.157207    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca" pod "558451a8-f2ef-435b-9699-04ebdec26e05" (UID: "558451a8-f2ef-435b-9699-04ebdec26e05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:07:56.157399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.157219    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\" (UniqueName: \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\") pod \"558451a8-f2ef-435b-9699-04ebdec26e05\" (UID: \"558451a8-f2ef-435b-9699-04ebdec26e05\") "
Nov 03 19:07:56.158123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.158098    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:56.159504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.159461    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca" pod "558451a8-f2ef-435b-9699-04ebdec26e05" (UID: "558451a8-f2ef-435b-9699-04ebdec26e05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:07:56.159884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.159863    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d" pod "558451a8-f2ef-435b-9699-04ebdec26e05" (UID: "558451a8-f2ef-435b-9699-04ebdec26e05"). InnerVolumeSpecName "kube-api-access-f8l4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:07:56.260412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.260375    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/558451a8-f2ef-435b-9699-04ebdec26e05-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:56.260412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.260413    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8l4d\" (UniqueName: \"kubernetes.io/projected/558451a8-f2ef-435b-9699-04ebdec26e05-kube-api-access-f8l4d\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:07:56.272017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.271984    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:07:56.890767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.890736    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"
Nov 03 19:07:56.891207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.890775    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4" event={"ID":"558451a8-f2ef-435b-9699-04ebdec26e05","Type":"ContainerDied","Data":"c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f"}
Nov 03 19:07:56.891207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.890857    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c11a2e2863f1e6041adec7e020a6bb9a59de9de74e366d4c5a4d410d600c9c1f"
Nov 03 19:07:56.945237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.945208    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:07:56.945871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.945857    4944 cpu_manager.go:414] "containerMap: removing container" podUID="558451a8-f2ef-435b-9699-04ebdec26e05" containerName="revoke-keys"
Nov 03 19:07:56.945943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.945872    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="558451a8-f2ef-435b-9699-04ebdec26e05" containerName="revoke-keys"
Nov 03 19:07:56.945943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.945903    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c1d18983-de37-422b-9928-78ff67798770" containerName="revoke-keys"
Nov 03 19:07:56.945943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.945908    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d18983-de37-422b-9928-78ff67798770" containerName="revoke-keys"
Nov 03 19:07:56.946037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.946013    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="558451a8-f2ef-435b-9699-04ebdec26e05" containerName="revoke-keys"
Nov 03 19:07:56.946037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.946023    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d18983-de37-422b-9928-78ff67798770" containerName="revoke-keys"
Nov 03 19:07:56.949715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.949697    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:56.998680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.998642    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 19:07:56.998680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.998642    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 19:07:56.998680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.998680    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 19:07:56.998922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:56.998742    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dqcwl\""
Nov 03 19:07:57.052097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.052062    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:07:57.072228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.072177    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.072410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.072335    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.072502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.072469    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.072743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.072704    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.073022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.073001    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.073103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.073039    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.073165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.073108    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v649l\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175251    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175417    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175447    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175479    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v649l\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175527    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175589    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.175699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.175648    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.176360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.176327    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.177924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.177899    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.178040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.177948    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.178040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.177961    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.178150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.178068    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.294008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.293973    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.340198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.340163    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v649l\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zflrw\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.562002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.561971    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:07:57.992629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.992602    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:07:57.996631 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:07:57.996603    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20f8a3ba_0552_493d_a49b_a4927964f245.slice/crio-80ea8e47066bbf4419d6cffe6c000fa6d7095ac30ad39f9495105bd46dc6a48c WatchSource:0}: Error finding container 80ea8e47066bbf4419d6cffe6c000fa6d7095ac30ad39f9495105bd46dc6a48c: Status 404 returned error can't find the container with id 80ea8e47066bbf4419d6cffe6c000fa6d7095ac30ad39f9495105bd46dc6a48c
Nov 03 19:07:57.998727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:57.998709    4944 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:07:58.906616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:07:58.906578    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" event={"ID":"20f8a3ba-0552-493d-a49b-a4927964f245","Type":"ContainerStarted","Data":"80ea8e47066bbf4419d6cffe6c000fa6d7095ac30ad39f9495105bd46dc6a48c"}
Nov 03 19:08:00.859465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:00.859427    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:02.049611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:02.049577    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"]
Nov 03 19:08:02.095519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:02.095488    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8add3de-52hgw"]
Nov 03 19:08:02.141028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:02.141001    4944 scope.go:122] "RemoveContainer" containerID="b7fb00dce7c523fdb9f1fcc0950c66e0de816bdab9bf2e9c7efbfe8f941f4215"
Nov 03 19:08:02.151402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:02.151383    4944 scope.go:122] "RemoveContainer" containerID="0f89bee52a7f72353ed5fe89b63eeafcdcb5e5f8d61fc579f7d4dedf3980ba9f"
Nov 03 19:08:02.163112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:02.163087    4944 scope.go:122] "RemoveContainer" containerID="76d77c6965ea2b8a3db0303934e145419846445c82fd62374a2d663ae03db7f9"
Nov 03 19:08:03.935820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:03.935773    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6873aa0-0e37-42a3-a0fe-37353d96edef" path="/var/lib/kubelet/pods/c6873aa0-0e37-42a3-a0fe-37353d96edef/volumes"
Nov 03 19:08:05.224976 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:08:05.224934    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:08:05.766597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:05.766560    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:07.357525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.357491    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"]
Nov 03 19:08:07.364124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.364107    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.414580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.414543    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"]
Nov 03 19:08:07.426644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.426608    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.426827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.426654    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6wb9\" (UniqueName: \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.426993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.426963    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.456103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.456046    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:07.456103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.456056    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:07.456103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.456079    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:07.458968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.458951    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:08:07.528877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.528843    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.529051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.528920    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.529051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.528944    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6wb9\" (UniqueName: \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.529765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.529746    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.532129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.532102    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.585925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.585896    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6wb9\" (UniqueName: \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\") pod \"maas-api-revoke-sub-ce701d90-9x9tv\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:07.679836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:07.679750    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:08.055381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.055347    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"]
Nov 03 19:08:08.059194 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:08.059168    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce87f234_5f06_4f98_87db_e2114177567c.slice/crio-5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3 WatchSource:0}: Error finding container 5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3: Status 404 returned error can't find the container with id 5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3
Nov 03 19:08:08.921811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.921183    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"]
Nov 03 19:08:08.928829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.928177    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:08.981943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.981850    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"]
Nov 03 19:08:08.996465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.996430    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv" event={"ID":"ce87f234-5f06-4f98-87db-e2114177567c","Type":"ContainerStarted","Data":"71bdfe0078163e9fbdb6c310e23725cf60960d7297b058c8e8ac862cca541c15"}
Nov 03 19:08:08.997526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:08.997498    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv" event={"ID":"ce87f234-5f06-4f98-87db-e2114177567c","Type":"ContainerStarted","Data":"5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3"}
Nov 03 19:08:09.061522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.059277    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2gdr\" (UniqueName: \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.061522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.059448    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.061522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.059591    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.212595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.212497    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2gdr\" (UniqueName: \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.214063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.214003    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.214251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.213022    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.214576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.214554    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.220207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.218858    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.249941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.235325    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv" podStartSLOduration=2.235305206 podStartE2EDuration="2.235305206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:09.167489498 +0000 UTC m=+1387.444843082" watchObservedRunningTime="2026-11-03 19:08:09.235305206 +0000 UTC m=+1387.512658792"
Nov 03 19:08:09.282815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.280893    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2gdr\" (UniqueName: \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\") pod \"maas-api-revoke-sub-488a6e86-ccx9h\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:09.566864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:09.553667    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:10.429871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.429205    4944 generic.go:356] "Generic (PLEG): container finished" podID="ce87f234-5f06-4f98-87db-e2114177567c" containerID="71bdfe0078163e9fbdb6c310e23725cf60960d7297b058c8e8ac862cca541c15" exitCode=0
Nov 03 19:08:10.494815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.484067    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv" event={"ID":"ce87f234-5f06-4f98-87db-e2114177567c","Type":"ContainerDied","Data":"71bdfe0078163e9fbdb6c310e23725cf60960d7297b058c8e8ac862cca541c15"}
Nov 03 19:08:10.494815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.484124    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"]
Nov 03 19:08:10.519814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.504077    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:10.669816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.655941    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:10.681225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.675920    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"]
Nov 03 19:08:10.681225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.676015    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.793876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.762851    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.822011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.821973    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77l6b\" (UniqueName: \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.822948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.822913    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.938112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.935023    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.938112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.935134    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77l6b\" (UniqueName: \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.938112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.935181    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.939592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.939252    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:10.995133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:10.991293    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:11.027052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.005955    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:11.027052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.006018    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.030001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.029972    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77l6b\" (UniqueName: \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\") pod \"maas-api-revoke-sub-ce7399fc-q9g7d\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:11.119274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.069971    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:11.121918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.121873    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.408584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.398904    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.473323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.463675    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.473323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.465155    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.473323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.472893    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.514686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.512938    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.514686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.513415    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:08:11.534321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.533930    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.604827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.579042    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.605310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.605194    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.781900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.712776    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:11.781900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.780263    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Nov 03 19:08:11.838297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:11.838249    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.036948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.036857    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.145582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.145530    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Nov 03 19:08:12.145867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.145838    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.403229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.403127    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.437915    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.439809    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.649606    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.649637    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"]
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.535952    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.439843    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.589383    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.536118    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": context deadline exceeded"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.591183    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:08:12.666872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.587928    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.673146    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.673216    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.673257    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.591154    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.591202    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.630080    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.677120    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:08:12.701301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.677169    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:08:12.743371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.715557    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:08:12.743371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.721202    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:08:12.760360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.758862    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:12.760360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.760169    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:08:12.810530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:12.810493    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 03 19:08:13.040424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.037427    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:13.040424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.039879    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:08:13.091849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.084152    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:08:13.135404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.132702    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:08:13.135715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.135694    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:08:13.157596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.157334    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:08:13.172679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.159541    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:08:13.172679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.162237    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:08:13.180817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.180064    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:08:13.188936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.188916    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:08:13.200152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.200132    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:08:13.210346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.210323    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:08:13.240435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.240313    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:08:13.254818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.253223    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"]
Nov 03 19:08:13.275318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.266951    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fb8f455-n5mff"]
Nov 03 19:08:13.285383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.285269    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h" event={"ID":"3097ba8e-57fe-47d0-bb93-7bc5daccee74","Type":"ContainerStarted","Data":"d28da363a545f4bc068eea66bbc9d688eb87a50795148c3453ecc7dda024d36a"}
Nov 03 19:08:13.354121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.354097    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:13.436260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.436228    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"]
Nov 03 19:08:13.496282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.496250    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\" (UniqueName: \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\") pod \"ce87f234-5f06-4f98-87db-e2114177567c\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") "
Nov 03 19:08:13.496546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.496523    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\") pod \"ce87f234-5f06-4f98-87db-e2114177567c\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") "
Nov 03 19:08:13.496618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.496601    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\") pod \"ce87f234-5f06-4f98-87db-e2114177567c\" (UID: \"ce87f234-5f06-4f98-87db-e2114177567c\") "
Nov 03 19:08:13.497287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.496976    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca" pod "ce87f234-5f06-4f98-87db-e2114177567c" (UID: "ce87f234-5f06-4f98-87db-e2114177567c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:13.497287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.497254    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:13.499162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.499138    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9" pod "ce87f234-5f06-4f98-87db-e2114177567c" (UID: "ce87f234-5f06-4f98-87db-e2114177567c"). InnerVolumeSpecName "kube-api-access-x6wb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:13.501170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.501133    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca" pod "ce87f234-5f06-4f98-87db-e2114177567c" (UID: "ce87f234-5f06-4f98-87db-e2114177567c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:13.602314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.602194    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce87f234-5f06-4f98-87db-e2114177567c-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:13.602314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.602229    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6wb9\" (UniqueName: \"kubernetes.io/projected/ce87f234-5f06-4f98-87db-e2114177567c-kube-api-access-x6wb9\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:13.946204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:13.946129    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="905ded08-5c6e-497d-b7c0-191d96f13421" path="/var/lib/kubelet/pods/905ded08-5c6e-497d-b7c0-191d96f13421/volumes"
Nov 03 19:08:14.296395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.296363    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"
Nov 03 19:08:14.297123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.297058    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv" event={"ID":"ce87f234-5f06-4f98-87db-e2114177567c","Type":"ContainerDied","Data":"5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3"}
Nov 03 19:08:14.297123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.297113    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a26108aa78e5239cfdf359b299090c0f2df8011ff6b8056e1c59dc645ac35b3"
Nov 03 19:08:14.299880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.299850    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d" event={"ID":"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54","Type":"ContainerStarted","Data":"017a179ac1d75fbdffa9004c37adfe045b18c3a37573af3c3ba19d63c332e819"}
Nov 03 19:08:14.300009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.299899    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d" event={"ID":"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54","Type":"ContainerStarted","Data":"0800e0e59a7212885b1f08dbe490f0e7a89053c3a0ac90a09f29290ab1619c62"}
Nov 03 19:08:14.302540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.302513    4944 generic.go:356] "Generic (PLEG): container finished" podID="3097ba8e-57fe-47d0-bb93-7bc5daccee74" containerID="795cf2ceec25d2a2fb29290f0b537f708c5b90f90d177bdcc71ec50cf6cb2766" exitCode=0
Nov 03 19:08:14.302667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:14.302620    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h" event={"ID":"3097ba8e-57fe-47d0-bb93-7bc5daccee74","Type":"ContainerDied","Data":"795cf2ceec25d2a2fb29290f0b537f708c5b90f90d177bdcc71ec50cf6cb2766"}
Nov 03 19:08:15.154340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.154308    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"]
Nov 03 19:08:15.155061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.155039    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ce87f234-5f06-4f98-87db-e2114177567c" containerName="revoke-keys"
Nov 03 19:08:15.155214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.155189    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce87f234-5f06-4f98-87db-e2114177567c" containerName="revoke-keys"
Nov 03 19:08:15.155426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.155409    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce87f234-5f06-4f98-87db-e2114177567c" containerName="revoke-keys"
Nov 03 19:08:15.159148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.159129    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.208550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.208518    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"]
Nov 03 19:08:15.223890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.223849    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:08:15.224011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.223925    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:08:15.232088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.232058    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.232273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.232211    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbmw\" (UniqueName: \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.232340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.232281    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.312691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.312659    4944 generic.go:356] "Generic (PLEG): container finished" podID="6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" containerID="017a179ac1d75fbdffa9004c37adfe045b18c3a37573af3c3ba19d63c332e819" exitCode=0
Nov 03 19:08:15.312854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.312750    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d" event={"ID":"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54","Type":"ContainerDied","Data":"017a179ac1d75fbdffa9004c37adfe045b18c3a37573af3c3ba19d63c332e819"}
Nov 03 19:08:15.334141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.334105    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.334319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.334233    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbmw\" (UniqueName: \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.334319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.334294    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.339667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.335594    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.386656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.386610    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.396357 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.396323    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbmw\" (UniqueName: \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\") pod \"maas-api-revoke-sub-6e013309-mpczx\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.400209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.399805    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:08:15.400209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.400044    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy" containerID="cri-o://6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7" gracePeriod=30
Nov 03 19:08:15.408948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.408871    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:08:15.409177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.409148    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy" containerID="cri-o://bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca" gracePeriod=30
Nov 03 19:08:15.418643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.418616    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:08:15.419661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.419636    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy" containerID="cri-o://3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0" gracePeriod=30
Nov 03 19:08:15.476732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.476698    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:15.592271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.592245    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:15.639689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.639658    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\" (UniqueName: \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\") pod \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") "
Nov 03 19:08:15.639892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.639721    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\") pod \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") "
Nov 03 19:08:15.639968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.639952    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\") pod \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\" (UID: \"3097ba8e-57fe-47d0-bb93-7bc5daccee74\") "
Nov 03 19:08:15.640389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.640351    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca" pod "3097ba8e-57fe-47d0-bb93-7bc5daccee74" (UID: "3097ba8e-57fe-47d0-bb93-7bc5daccee74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:15.640775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.640751    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:15.641867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.641840    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca" pod "3097ba8e-57fe-47d0-bb93-7bc5daccee74" (UID: "3097ba8e-57fe-47d0-bb93-7bc5daccee74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:15.641985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.641954    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr" pod "3097ba8e-57fe-47d0-bb93-7bc5daccee74" (UID: "3097ba8e-57fe-47d0-bb93-7bc5daccee74"). InnerVolumeSpecName "kube-api-access-s2gdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:15.742311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.742267    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2gdr\" (UniqueName: \"kubernetes.io/projected/3097ba8e-57fe-47d0-bb93-7bc5daccee74-kube-api-access-s2gdr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:15.742311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.742302    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3097ba8e-57fe-47d0-bb93-7bc5daccee74-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:15.894101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:15.894066    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"]
Nov 03 19:08:15.896008 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:15.895973    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a47c01_0678_439c_8a1c_b96a5828b158.slice/crio-f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e WatchSource:0}: Error finding container f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e: Status 404 returned error can't find the container with id f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e
Nov 03 19:08:16.324525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.324498    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"
Nov 03 19:08:16.324525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.324505    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h" event={"ID":"3097ba8e-57fe-47d0-bb93-7bc5daccee74","Type":"ContainerDied","Data":"d28da363a545f4bc068eea66bbc9d688eb87a50795148c3453ecc7dda024d36a"}
Nov 03 19:08:16.324964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.324547    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28da363a545f4bc068eea66bbc9d688eb87a50795148c3453ecc7dda024d36a"
Nov 03 19:08:16.326311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.326281    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" event={"ID":"20f8a3ba-0552-493d-a49b-a4927964f245","Type":"ContainerStarted","Data":"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"}
Nov 03 19:08:16.326311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.326331    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:08:16.328163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.328142    4944 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zflrw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:08:16.328273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.328182    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:08:16.328420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.328398    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx" event={"ID":"b4a47c01-0678-439c-8a1c-b96a5828b158","Type":"ContainerStarted","Data":"0528e8b4f3706c1af950723cb259dfa18c304406113472eaf7f9a41c26daedd0"}
Nov 03 19:08:16.328508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.328452    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx" event={"ID":"b4a47c01-0678-439c-8a1c-b96a5828b158","Type":"ContainerStarted","Data":"f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e"}
Nov 03 19:08:16.456466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.456398    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" podStartSLOduration=3.234305649 podStartE2EDuration="20.456378765s" totalImagesPullingTime="17.222073116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:16.445905542 +0000 UTC m=+1394.723259126" watchObservedRunningTime="2026-11-03 19:08:16.456378765 +0000 UTC m=+1394.733732349"
Nov 03 19:08:16.516100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.516070    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:16.565432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.565395    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\") pod \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") "
Nov 03 19:08:16.565432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.565435    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\" (UniqueName: \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\") pod \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") "
Nov 03 19:08:16.565688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.565493    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\") pod \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\" (UID: \"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54\") "
Nov 03 19:08:16.566483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.566448    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca" pod "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" (UID: "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:16.568843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.568252    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca" pod "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" (UID: "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:16.568843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.568489    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b" pod "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" (UID: "6af61ef8-fa64-41e5-8e6a-f7c1c3157f54"). InnerVolumeSpecName "kube-api-access-77l6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:16.569040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.568851    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx" podStartSLOduration=1.568835826 podStartE2EDuration="1.568835826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:16.560401381 +0000 UTC m=+1394.837754970" watchObservedRunningTime="2026-11-03 19:08:16.568835826 +0000 UTC m=+1394.846189410"
Nov 03 19:08:16.668170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.668078    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:16.668170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.668116    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77l6b\" (UniqueName: \"kubernetes.io/projected/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-kube-api-access-77l6b\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:16.668170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:16.668132    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:17.341351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.341318    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"
Nov 03 19:08:17.341351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.341338    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d" event={"ID":"6af61ef8-fa64-41e5-8e6a-f7c1c3157f54","Type":"ContainerDied","Data":"0800e0e59a7212885b1f08dbe490f0e7a89053c3a0ac90a09f29290ab1619c62"}
Nov 03 19:08:17.341880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.341389    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0800e0e59a7212885b1f08dbe490f0e7a89053c3a0ac90a09f29290ab1619c62"
Nov 03 19:08:17.343231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.343204    4944 generic.go:356] "Generic (PLEG): container finished" podID="b4a47c01-0678-439c-8a1c-b96a5828b158" containerID="0528e8b4f3706c1af950723cb259dfa18c304406113472eaf7f9a41c26daedd0" exitCode=0
Nov 03 19:08:17.343364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.343284    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx" event={"ID":"b4a47c01-0678-439c-8a1c-b96a5828b158","Type":"ContainerDied","Data":"0528e8b4f3706c1af950723cb259dfa18c304406113472eaf7f9a41c26daedd0"}
Nov 03 19:08:17.344268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.344252    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:08:17.733144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:17.733054    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:18.527661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.527633    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:18.598489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.598458    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\") pod \"b4a47c01-0678-439c-8a1c-b96a5828b158\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") "
Nov 03 19:08:18.598648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.598497    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\" (UniqueName: \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\") pod \"b4a47c01-0678-439c-8a1c-b96a5828b158\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") "
Nov 03 19:08:18.598817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.598777    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\") pod \"b4a47c01-0678-439c-8a1c-b96a5828b158\" (UID: \"b4a47c01-0678-439c-8a1c-b96a5828b158\") "
Nov 03 19:08:18.599106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.599081    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca" pod "b4a47c01-0678-439c-8a1c-b96a5828b158" (UID: "b4a47c01-0678-439c-8a1c-b96a5828b158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:18.599367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.599351    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:18.601132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.601108    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw" pod "b4a47c01-0678-439c-8a1c-b96a5828b158" (UID: "b4a47c01-0678-439c-8a1c-b96a5828b158"). InnerVolumeSpecName "kube-api-access-tbbmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:18.601228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.601107    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca" pod "b4a47c01-0678-439c-8a1c-b96a5828b158" (UID: "b4a47c01-0678-439c-8a1c-b96a5828b158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:18.701293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.701260    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4a47c01-0678-439c-8a1c-b96a5828b158-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:18.701293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:18.701288    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbbmw\" (UniqueName: \"kubernetes.io/projected/b4a47c01-0678-439c-8a1c-b96a5828b158-kube-api-access-tbbmw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:19.363107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:19.363066    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx" event={"ID":"b4a47c01-0678-439c-8a1c-b96a5828b158","Type":"ContainerDied","Data":"f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e"}
Nov 03 19:08:19.363271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:19.363123    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f98a55f424290ea00bd18e79f72d2d185ffa1b3d708f9981d2f0ef907c87019e"
Nov 03 19:08:19.363271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:19.363082    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"
Nov 03 19:08:19.909979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:19.909945    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:20.157055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.157022    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:20.737218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.737188    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:08:20.768960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.762046    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:08:20.768960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.768086    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:08:20.831088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831054    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.831271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831115    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.831271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831174    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.831271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831211    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.831271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831228    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.831271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831264    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.831557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831324    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.831628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831550    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.831682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831648    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:20.831803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.831767    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.832491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832458    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.832582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832532    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.832582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832563    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.832690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832599    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.832690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832637    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.832845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832687    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.832845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832746    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.832845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832825    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.833027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832865    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.833027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832901    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.833027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832954    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.832979    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833001    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833069    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833127    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833167    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833209    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\") pod \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\" (UID: \"7e2ca6f6-e658-40bb-9a45-83ff8205c71e\") "
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833225    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833254    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833294    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833303    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833363    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.833712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833562    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833569    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833662    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:08:20.833712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833697    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833718    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.833935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833838    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.834029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.833933    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.834029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834002    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\") pod \"361411e7-0710-4f0f-8107-2cdd8c83af3b\" (UID: \"361411e7-0710-4f0f-8107-2cdd8c83af3b\") "
Nov 03 19:08:20.834029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834025    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\") pod \"8d4d849c-0681-411a-81c6-aef900ec09b2\" (UID: \"8d4d849c-0681-411a-81c6-aef900ec09b2\") "
Nov 03 19:08:20.834681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834019    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834045    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834419    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834567    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834739    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834756    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834764    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834772    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.834784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834781    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834829    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834844    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834856    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834869    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834881    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834890    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834898    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834906    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.835262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.834914    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4d849c-0681-411a-81c6-aef900ec09b2-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.836066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836040    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.836654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836621    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:08:20.836754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836668    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.836868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836766    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "kube-api-access-qfdsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.836868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836811    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt" pod "8d4d849c-0681-411a-81c6-aef900ec09b2" (UID: "8d4d849c-0681-411a-81c6-aef900ec09b2"). InnerVolumeSpecName "kube-api-access-cxhlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.837004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.836915    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:08:20.837097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.837082    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo" pod "361411e7-0710-4f0f-8107-2cdd8c83af3b" (UID: "361411e7-0710-4f0f-8107-2cdd8c83af3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:08:20.837252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.837231    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "kube-api-access-j9sdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.838200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.838175    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token" pod "7e2ca6f6-e658-40bb-9a45-83ff8205c71e" (UID: "7e2ca6f6-e658-40bb-9a45-83ff8205c71e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:20.936104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936072    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936100    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxhlt\" (UniqueName: \"kubernetes.io/projected/8d4d849c-0681-411a-81c6-aef900ec09b2-kube-api-access-cxhlt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936112    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdsl\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-kube-api-access-qfdsl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936122    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936131    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9sdl\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-kube-api-access-j9sdl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936139    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/361411e7-0710-4f0f-8107-2cdd8c83af3b-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936147    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4d849c-0681-411a-81c6-aef900ec09b2-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936156    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936164    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4d849c-0681-411a-81c6-aef900ec09b2-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936172    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/361411e7-0710-4f0f-8107-2cdd8c83af3b-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936180    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936189    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:20.936494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:20.936197    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e2ca6f6-e658-40bb-9a45-83ff8205c71e-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:21.393165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.393130    4944 generic.go:356] "Generic (PLEG): container finished" podID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerID="bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca" exitCode=0
Nov 03 19:08:21.393346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.393196    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"
Nov 03 19:08:21.393346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.393210    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" event={"ID":"361411e7-0710-4f0f-8107-2cdd8c83af3b","Type":"ContainerDied","Data":"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"}
Nov 03 19:08:21.393346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.393297    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc" event={"ID":"361411e7-0710-4f0f-8107-2cdd8c83af3b","Type":"ContainerDied","Data":"daa0bdb6f39257d8d565db3fdada0be3d68ab9ecdb7907f7e8a93fffa772818d"}
Nov 03 19:08:21.393346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.393331    4944 scope.go:122] "RemoveContainer" containerID="bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"
Nov 03 19:08:21.394696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.394672    4944 generic.go:356] "Generic (PLEG): container finished" podID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerID="6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7" exitCode=0
Nov 03 19:08:21.394824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.394725    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"
Nov 03 19:08:21.394824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.394751    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" event={"ID":"8d4d849c-0681-411a-81c6-aef900ec09b2","Type":"ContainerDied","Data":"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"}
Nov 03 19:08:21.394956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.394827    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h" event={"ID":"8d4d849c-0681-411a-81c6-aef900ec09b2","Type":"ContainerDied","Data":"6a99029fbaea16cc207c41170aed8d7e5b254ee40b5f988dcf2ba9b5ef9587f9"}
Nov 03 19:08:21.397004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.396967    4944 generic.go:356] "Generic (PLEG): container finished" podID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerID="3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0" exitCode=0
Nov 03 19:08:21.397141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.397029    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"
Nov 03 19:08:21.397241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.397192    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" event={"ID":"7e2ca6f6-e658-40bb-9a45-83ff8205c71e","Type":"ContainerDied","Data":"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"}
Nov 03 19:08:21.397241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.397231    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl" event={"ID":"7e2ca6f6-e658-40bb-9a45-83ff8205c71e","Type":"ContainerDied","Data":"75a3614e0663b2fa4f0603918a4ea590a83b97a90f92a7ba0030dbd9a76fad82"}
Nov 03 19:08:21.407454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.406408    4944 scope.go:122] "RemoveContainer" containerID="bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"
Nov 03 19:08:21.407454 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:08:21.406726    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca\": container with ID starting with bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca not found: ID does not exist" containerID="bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"
Nov 03 19:08:21.407454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.406759    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca"} err="failed to get container status \"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca\": rpc error: code = NotFound desc = could not find container \"bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca\": container with ID starting with bcb4b89c45e2a0fdffbc9e292c8422a402cba4b6d4368133309c9298ffd3feca not found: ID does not exist"
Nov 03 19:08:21.407454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.406783    4944 scope.go:122] "RemoveContainer" containerID="6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"
Nov 03 19:08:21.418577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.418557    4944 scope.go:122] "RemoveContainer" containerID="6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"
Nov 03 19:08:21.418948 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:08:21.418925    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7\": container with ID starting with 6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7 not found: ID does not exist" containerID="6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"
Nov 03 19:08:21.419030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.418956    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7"} err="failed to get container status \"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7\": rpc error: code = NotFound desc = could not find container \"6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7\": container with ID starting with 6f54c45b40bad90f77fc22ea7e10e72cbb0a37ffd1f601f1770cbd1ae7989ce7 not found: ID does not exist"
Nov 03 19:08:21.419030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.418979    4944 scope.go:122] "RemoveContainer" containerID="3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"
Nov 03 19:08:21.444834 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.444815    4944 scope.go:122] "RemoveContainer" containerID="3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"
Nov 03 19:08:21.445143 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:08:21.445116    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0\": container with ID starting with 3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0 not found: ID does not exist" containerID="3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"
Nov 03 19:08:21.445241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.445165    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0"} err="failed to get container status \"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0\": rpc error: code = NotFound desc = could not find container \"3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0\": container with ID starting with 3f0875c4cdfa81c86c6031ad58f1ea6e3516b481b302b5181773d8fda616a7d0 not found: ID does not exist"
Nov 03 19:08:21.551073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.551029    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:08:21.556498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.556472    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-tf8hc"]
Nov 03 19:08:21.662416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.662324    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:08:21.667879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.667843    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6kcl"]
Nov 03 19:08:21.768862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.768803    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:08:21.773233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.773210    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pd82h"]
Nov 03 19:08:21.937008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.936936    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" path="/var/lib/kubelet/pods/361411e7-0710-4f0f-8107-2cdd8c83af3b/volumes"
Nov 03 19:08:21.937476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.937460    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" path="/var/lib/kubelet/pods/7e2ca6f6-e658-40bb-9a45-83ff8205c71e/volumes"
Nov 03 19:08:21.937915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:21.937902    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" path="/var/lib/kubelet/pods/8d4d849c-0681-411a-81c6-aef900ec09b2/volumes"
Nov 03 19:08:22.300492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.300453    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:22.781840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.781777    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"]
Nov 03 19:08:22.782501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782484    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" containerName="revoke-keys"
Nov 03 19:08:22.782552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782502    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" containerName="revoke-keys"
Nov 03 19:08:22.782552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782534    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy"
Nov 03 19:08:22.782552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782539    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782555    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b4a47c01-0678-439c-8a1c-b96a5828b158" containerName="revoke-keys"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782560    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a47c01-0678-439c-8a1c-b96a5828b158" containerName="revoke-keys"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782569    4944 cpu_manager.go:414] "containerMap: removing container" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782574    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782586    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782591    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782601    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3097ba8e-57fe-47d0-bb93-7bc5daccee74" containerName="revoke-keys"
Nov 03 19:08:22.782656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782608    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3097ba8e-57fe-47d0-bb93-7bc5daccee74" containerName="revoke-keys"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782700    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a47c01-0678-439c-8a1c-b96a5828b158" containerName="revoke-keys"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782709    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="361411e7-0710-4f0f-8107-2cdd8c83af3b" containerName="istio-proxy"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782719    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e2ca6f6-e658-40bb-9a45-83ff8205c71e" containerName="istio-proxy"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782726    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3097ba8e-57fe-47d0-bb93-7bc5daccee74" containerName="revoke-keys"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782734    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" containerName="revoke-keys"
Nov 03 19:08:22.782922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.782744    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4d849c-0681-411a-81c6-aef900ec09b2" containerName="istio-proxy"
Nov 03 19:08:22.788203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.788182    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.840360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.840326    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"]
Nov 03 19:08:22.861522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.861495    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.861766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.861734    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.861850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.861784    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwr6s\" (UniqueName: \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.879950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.879922    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:22.880084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.879986    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:22.880084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.880014    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:22.882651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.882629    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:08:22.963262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.963229    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.963633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.963270    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwr6s\" (UniqueName: \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.963633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.963346    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.963911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.963889    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:22.965615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:22.965596    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:23.020512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:23.020479    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwr6s\" (UniqueName: \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\") pod \"maas-api-revoke-sub-dcd98f58-24pdw\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:23.102336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:23.102257    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:23.482156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:23.482076    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"]
Nov 03 19:08:23.486430 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:23.486401    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a404563_5183_452e_8989_9f5ffdff1309.slice/crio-4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02 WatchSource:0}: Error finding container 4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02: Status 404 returned error can't find the container with id 4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02
Nov 03 19:08:23.631519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:23.631480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:24.425701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:24.425669    4944 generic.go:356] "Generic (PLEG): container finished" podID="0a404563-5183-452e-8989-9f5ffdff1309" containerID="4af5ff372fbaea42c226f55b734dd77725b93006a8945f4b77e35259128aaba8" exitCode=0
Nov 03 19:08:24.426090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:24.425760    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw" event={"ID":"0a404563-5183-452e-8989-9f5ffdff1309","Type":"ContainerDied","Data":"4af5ff372fbaea42c226f55b734dd77725b93006a8945f4b77e35259128aaba8"}
Nov 03 19:08:24.426090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:24.425858    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw" event={"ID":"0a404563-5183-452e-8989-9f5ffdff1309","Type":"ContainerStarted","Data":"4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02"}
Nov 03 19:08:25.584597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.584571    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:25.701241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.701161    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\" (UniqueName: \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\") pod \"0a404563-5183-452e-8989-9f5ffdff1309\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") "
Nov 03 19:08:25.701241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.701222    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\") pod \"0a404563-5183-452e-8989-9f5ffdff1309\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") "
Nov 03 19:08:25.701552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.701534    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\") pod \"0a404563-5183-452e-8989-9f5ffdff1309\" (UID: \"0a404563-5183-452e-8989-9f5ffdff1309\") "
Nov 03 19:08:25.702420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.702387    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca" pod "0a404563-5183-452e-8989-9f5ffdff1309" (UID: "0a404563-5183-452e-8989-9f5ffdff1309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:25.704352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.704324    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s" pod "0a404563-5183-452e-8989-9f5ffdff1309" (UID: "0a404563-5183-452e-8989-9f5ffdff1309"). InnerVolumeSpecName "kube-api-access-mwr6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:25.704631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.704601    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca" pod "0a404563-5183-452e-8989-9f5ffdff1309" (UID: "0a404563-5183-452e-8989-9f5ffdff1309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:25.803079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.803040    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwr6s\" (UniqueName: \"kubernetes.io/projected/0a404563-5183-452e-8989-9f5ffdff1309-kube-api-access-mwr6s\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.803079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.803073    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.803079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.803083    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a404563-5183-452e-8989-9f5ffdff1309-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:25.833719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.833687    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"]
Nov 03 19:08:25.834370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.834355    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0a404563-5183-452e-8989-9f5ffdff1309" containerName="revoke-keys"
Nov 03 19:08:25.834416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.834373    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a404563-5183-452e-8989-9f5ffdff1309" containerName="revoke-keys"
Nov 03 19:08:25.834494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.834485    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a404563-5183-452e-8989-9f5ffdff1309" containerName="revoke-keys"
Nov 03 19:08:25.838263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.838244    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:25.888989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:25.888960    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"]
Nov 03 19:08:26.006532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.006494    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.006721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.006550    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.006721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.006703    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4zg\" (UniqueName: \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.107906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.107874    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.108091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.107923    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.108091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.108032    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4zg\" (UniqueName: \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.108589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.108570    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.110332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.110314    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.165601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.165571    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4zg\" (UniqueName: \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\") pod \"maas-api-revoke-sub-fbcffd8a-8krx8\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.443667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.443596    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"
Nov 03 19:08:26.443667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.443646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw" event={"ID":"0a404563-5183-452e-8989-9f5ffdff1309","Type":"ContainerDied","Data":"4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02"}
Nov 03 19:08:26.443870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.443698    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4114497dbda6b406f7d7429aed551951c2846c348d08c2b5e3b7d45a7f9a8e02"
Nov 03 19:08:26.453682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.453663    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:26.833722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.833688    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"]
Nov 03 19:08:26.835965 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:26.835874    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e677ee_8b43_4150_b193_6e7d98b53bb2.slice/crio-abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8 WatchSource:0}: Error finding container abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8: Status 404 returned error can't find the container with id abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8
Nov 03 19:08:26.846849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:26.846818    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:27.453524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:27.453438    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8" event={"ID":"a3e677ee-8b43-4150-b193-6e7d98b53bb2","Type":"ContainerStarted","Data":"e44c1bee26b78c9115402e4429f83bd089627cb73c3149a9c5dbbccdf99263c0"}
Nov 03 19:08:27.453524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:27.453495    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8" event={"ID":"a3e677ee-8b43-4150-b193-6e7d98b53bb2","Type":"ContainerStarted","Data":"abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8"}
Nov 03 19:08:27.561468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:27.561415    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8" podStartSLOduration=2.561402812 podStartE2EDuration="2.561402812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:27.560775537 +0000 UTC m=+1405.838129120" watchObservedRunningTime="2026-11-03 19:08:27.561402812 +0000 UTC m=+1405.838756441"
Nov 03 19:08:28.361060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.361025    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:08:28.365488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.365470    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.466051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.466013    4944 generic.go:356] "Generic (PLEG): container finished" podID="a3e677ee-8b43-4150-b193-6e7d98b53bb2" containerID="e44c1bee26b78c9115402e4429f83bd089627cb73c3149a9c5dbbccdf99263c0" exitCode=0
Nov 03 19:08:28.466243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.466102    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8" event={"ID":"a3e677ee-8b43-4150-b193-6e7d98b53bb2","Type":"ContainerDied","Data":"e44c1bee26b78c9115402e4429f83bd089627cb73c3149a9c5dbbccdf99263c0"}
Nov 03 19:08:28.546188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546190    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546304    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scxlj\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546350    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546429    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546477    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546497    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546600    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.546694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.546670    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.564666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.564620    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:08:28.648125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648050    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648119    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648253    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648283    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648354    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scxlj\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648392    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648434    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648476    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648489    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648529    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648612    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.648942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.648917    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.649060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.649033    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.649177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.649145    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.651176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.651150    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.651389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.651367    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.705473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.705443    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.705673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.705650    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scxlj\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\") pod \"e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") " pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:28.979643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:28.979551    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:29.409649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.409614    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:08:29.449146 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:29.449107    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c44d648_90d2_4f52_8008_ff18d660ef15.slice/crio-1389d22e5355cbd8ca229cc4b5dba4cd501db00a609cc2b7cb745fb4d4330eeb WatchSource:0}: Error finding container 1389d22e5355cbd8ca229cc4b5dba4cd501db00a609cc2b7cb745fb4d4330eeb: Status 404 returned error can't find the container with id 1389d22e5355cbd8ca229cc4b5dba4cd501db00a609cc2b7cb745fb4d4330eeb
Nov 03 19:08:29.451858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.451822    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:08:29.451953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.451894    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:08:29.451953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.451923    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:08:29.475659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.475631    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" event={"ID":"7c44d648-90d2-4f52-8008-ff18d660ef15","Type":"ContainerStarted","Data":"1389d22e5355cbd8ca229cc4b5dba4cd501db00a609cc2b7cb745fb4d4330eeb"}
Nov 03 19:08:29.628517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.628493    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:29.662863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.662774    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\") pod \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") "
Nov 03 19:08:29.663008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.662877    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\" (UniqueName: \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\") pod \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") "
Nov 03 19:08:29.663057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.663012    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\") pod \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\" (UID: \"a3e677ee-8b43-4150-b193-6e7d98b53bb2\") "
Nov 03 19:08:29.663417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.663381    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca" pod "a3e677ee-8b43-4150-b193-6e7d98b53bb2" (UID: "a3e677ee-8b43-4150-b193-6e7d98b53bb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:29.663645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.663627    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:29.664749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.664727    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca" pod "a3e677ee-8b43-4150-b193-6e7d98b53bb2" (UID: "a3e677ee-8b43-4150-b193-6e7d98b53bb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:29.664890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.664868    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg" pod "a3e677ee-8b43-4150-b193-6e7d98b53bb2" (UID: "a3e677ee-8b43-4150-b193-6e7d98b53bb2"). InnerVolumeSpecName "kube-api-access-bw4zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:29.765535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.765493    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3e677ee-8b43-4150-b193-6e7d98b53bb2-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:29.765535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:29.765535    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4zg\" (UniqueName: \"kubernetes.io/projected/a3e677ee-8b43-4150-b193-6e7d98b53bb2-kube-api-access-bw4zg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:30.485152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.485117    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"
Nov 03 19:08:30.485152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.485134    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8" event={"ID":"a3e677ee-8b43-4150-b193-6e7d98b53bb2","Type":"ContainerDied","Data":"abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8"}
Nov 03 19:08:30.485657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.485222    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc2c809b3e403480ccc57ad1f3b7f5b6337ac019906c9f7d56b761e3de7beb8"
Nov 03 19:08:30.487624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.487238    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" event={"ID":"7c44d648-90d2-4f52-8008-ff18d660ef15","Type":"ContainerStarted","Data":"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"}
Nov 03 19:08:30.602561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.602509    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podStartSLOduration=2.602495734 podStartE2EDuration="2.602495734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:30.599927466 +0000 UTC m=+1408.877281051" watchObservedRunningTime="2026-11-03 19:08:30.602495734 +0000 UTC m=+1408.879849382"
Nov 03 19:08:30.980145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.980113    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:30.981860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.981821    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:30.981995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:30.981899    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:31.338297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.338263    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"]
Nov 03 19:08:31.338981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.338965    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a3e677ee-8b43-4150-b193-6e7d98b53bb2" containerName="revoke-keys"
Nov 03 19:08:31.339049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.338982    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e677ee-8b43-4150-b193-6e7d98b53bb2" containerName="revoke-keys"
Nov 03 19:08:31.339099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.339090    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e677ee-8b43-4150-b193-6e7d98b53bb2" containerName="revoke-keys"
Nov 03 19:08:31.343049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.343027    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.387897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.387865    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.388050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.387993    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bttvn\" (UniqueName: \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.388163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.388146    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.392145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.392116    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:31.395010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.394981    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"]
Nov 03 19:08:31.435845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.430131    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:31.435845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.430252    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:08:31.435845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.430306    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:31.494331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.494292    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.495087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.495065    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.496224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.496074    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.496780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.496757    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bttvn\" (UniqueName: \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.498721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.497647    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.544238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.544204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:31.554209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.554176    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bttvn\" (UniqueName: \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\") pod \"maas-api-revoke-sub-23484ec6-wnbzw\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.657601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.657512    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:31.979986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.979904    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:31.979986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:31.979962    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:32.077117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.077085    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"]
Nov 03 19:08:32.078059 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:32.078028    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880ab551_8781_47f4_a958_f9dbd5316d53.slice/crio-26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4 WatchSource:0}: Error finding container 26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4: Status 404 returned error can't find the container with id 26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4
Nov 03 19:08:32.556366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.556335    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw" event={"ID":"880ab551-8781-47f4-a958-f9dbd5316d53","Type":"ContainerStarted","Data":"3e0c1d7d8195ae2b0b66694c593bf36754d75387a3be71acddd61fd2c30e0430"}
Nov 03 19:08:32.556833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.556390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw" event={"ID":"880ab551-8781-47f4-a958-f9dbd5316d53","Type":"ContainerStarted","Data":"26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4"}
Nov 03 19:08:32.669551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.669437    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw" podStartSLOduration=1.66941847 podStartE2EDuration="1.66941847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:32.665527046 +0000 UTC m=+1410.942880630" watchObservedRunningTime="2026-11-03 19:08:32.66941847 +0000 UTC m=+1410.946772053"
Nov 03 19:08:32.979914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.979837    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:32.979914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:32.979892    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:33.600651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:33.600616    4944 generic.go:356] "Generic (PLEG): container finished" podID="880ab551-8781-47f4-a958-f9dbd5316d53" containerID="3e0c1d7d8195ae2b0b66694c593bf36754d75387a3be71acddd61fd2c30e0430" exitCode=0
Nov 03 19:08:33.601105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:33.600694    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw" event={"ID":"880ab551-8781-47f4-a958-f9dbd5316d53","Type":"ContainerDied","Data":"3e0c1d7d8195ae2b0b66694c593bf36754d75387a3be71acddd61fd2c30e0430"}
Nov 03 19:08:33.980863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:33.980355    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:33.980863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:33.980425    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:34.986290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:34.985748    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:34.986290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:34.985834    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:35.141886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.140725    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.142088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.141902    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.205753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.204994    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.269813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.252887    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.269813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.254649    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.535868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.516542    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:35.746008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:35.745949    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:08:36.162862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.137991    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:36.162862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.138068    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:36.182864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.167997    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:08:36.182864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.171891    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.182864 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:08:36.177975    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:08:36.182864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.178117    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:08:36.219815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.183302    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.219815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.185887    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.219815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.217162    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.219815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.218251    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.254819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.240200    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:08:36.254819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.240265    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:08:36.254819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.240299    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:08:36.254819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.247009    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.255819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.255416    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.255819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.255673    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:08:36.268815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.256602    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:08:36.268815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.256658    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:08:36.570541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.570502    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.581226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.581197    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:08:36.753284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.752264    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:36.753284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.752408    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:08:36.782820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.763838    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:36.830815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.820530    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\") pod \"880ab551-8781-47f4-a958-f9dbd5316d53\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") "
Nov 03 19:08:36.830815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.820675    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\") pod \"880ab551-8781-47f4-a958-f9dbd5316d53\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") "
Nov 03 19:08:36.830815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.820750    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\" (UniqueName: \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\") pod \"880ab551-8781-47f4-a958-f9dbd5316d53\" (UID: \"880ab551-8781-47f4-a958-f9dbd5316d53\") "
Nov 03 19:08:36.837491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.837456    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca" pod "880ab551-8781-47f4-a958-f9dbd5316d53" (UID: "880ab551-8781-47f4-a958-f9dbd5316d53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:36.886996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.886956    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn" pod "880ab551-8781-47f4-a958-f9dbd5316d53" (UID: "880ab551-8781-47f4-a958-f9dbd5316d53"). InnerVolumeSpecName "kube-api-access-bttvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:36.966815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.932089    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:36.966815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.932130    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bttvn\" (UniqueName: \"kubernetes.io/projected/880ab551-8781-47f4-a958-f9dbd5316d53-kube-api-access-bttvn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:37.001814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:36.998506    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:37.002135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.002082    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:37.046217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.046100    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:37.187718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.187682    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:37.188670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.188634    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.191953    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca" pod "880ab551-8781-47f4-a958-f9dbd5316d53" (UID: "880ab551-8781-47f4-a958-f9dbd5316d53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:37.220095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.192449    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:37.220518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.192527    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:08:37.220767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.220751    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:08:37.221312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.221271    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:37.221568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.221544    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.222043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.222011    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:08:37.222229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.222209    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:08:37.462865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.426313    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.936297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.569898    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.936297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.569951    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.936297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.569996    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:08:37.936297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.592354    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:37.960660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.938758    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.006821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:37.595779    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:08:38.165028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.164981    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.165416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.165392    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:08:38.194124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.188922    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.241065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.216339    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880ab551-8781-47f4-a958-f9dbd5316d53-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.281150    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.281218    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.284166    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.291918    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.293082    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:08:38.325382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.293982    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.347779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.347719    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.684842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.630242    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"
Nov 03 19:08:38.733814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.711588    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw" event={"ID":"880ab551-8781-47f4-a958-f9dbd5316d53","Type":"ContainerDied","Data":"26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4"}
Nov 03 19:08:38.743083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.743035    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26031c9defb07c10ea40c685c1f2b2c6f0df70a99edf2b7666a6a8bca194c1f4"
Nov 03 19:08:38.781825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.781770    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:08:38.782109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.782077    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:38.961516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.961422    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.037776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.979942    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.038303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.979980    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.043849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.980452    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.044439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.980478    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.044832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:38.980759    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:39.045094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.045056    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:39.207349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.207309    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:08:39.283840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.264263    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:39.308678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.308646    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:08:39.321814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.314837    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:39.321814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.314887    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:39.321814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.314912    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:39.407809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.406985    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:08:39.426808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.423661    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:08:39.456122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.456093    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:08:39.457731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.457712    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:08:39.467844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.467822    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:08:39.471599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.471579    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:08:39.495617 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.495554    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:08:39.506268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.505142    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:08:39.521138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.520882    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:08:39.539094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.538784    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:08:39.541140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.540686    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:08:39.564485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.564457    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:08:39.982668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.982577    4944 patch_prober.go:29] interesting pod/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Nov 03 19:08:39.982859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:39.982666    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Nov 03 19:08:40.985759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:40.985706    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:40.986267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:40.986213    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:40.986974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:40.986951    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:08:41.944215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:41.944184    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:44.438046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.438011    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"]
Nov 03 19:08:44.439111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.439094    4944 cpu_manager.go:414] "containerMap: removing container" podUID="880ab551-8781-47f4-a958-f9dbd5316d53" containerName="revoke-keys"
Nov 03 19:08:44.439172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.439112    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880ab551-8781-47f4-a958-f9dbd5316d53" containerName="revoke-keys"
Nov 03 19:08:44.439248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.439236    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="880ab551-8781-47f4-a958-f9dbd5316d53" containerName="revoke-keys"
Nov 03 19:08:44.450394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.444597    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.496463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.496428    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"]
Nov 03 19:08:44.535744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.535718    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:08:44.535744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.535718    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:08:44.535972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.535773    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:08:44.537333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.537309    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.537471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.537452    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8l6\" (UniqueName: \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.537605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.537584    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.541953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.541932    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:08:44.638785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.638751    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.638985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.638851    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.639032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.639009    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8l6\" (UniqueName: \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.639459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.639422    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.641123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.641103    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.696294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.696226    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8l6\" (UniqueName: \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\") pod \"maas-api-revoke-sub-1c62ba4d-vwqd6\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:44.763507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:44.763476    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:45.142860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:45.142825    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"]
Nov 03 19:08:45.146879 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:45.146840    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b103112_1657_4877_b0bb_d76e9c68304a.slice/crio-0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60 WatchSource:0}: Error finding container 0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60: Status 404 returned error can't find the container with id 0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60
Nov 03 19:08:45.702477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:45.702390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6" event={"ID":"1b103112-1657-4877-b0bb-d76e9c68304a","Type":"ContainerStarted","Data":"fdbd9d49fd777ef4a8d3e90a66a99c57c6293f86cb95415bf12a1a7134ef4880"}
Nov 03 19:08:45.702477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:45.702452    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6" event={"ID":"1b103112-1657-4877-b0bb-d76e9c68304a","Type":"ContainerStarted","Data":"0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60"}
Nov 03 19:08:45.812068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:45.812016    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6" podStartSLOduration=1.812000577 podStartE2EDuration="1.812000577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:45.809216776 +0000 UTC m=+1424.086570359" watchObservedRunningTime="2026-11-03 19:08:45.812000577 +0000 UTC m=+1424.089354199"
Nov 03 19:08:46.462581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.462494    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"]
Nov 03 19:08:46.466993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.466970    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.521837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.521807    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"]
Nov 03 19:08:46.564592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.564558    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.564775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.564691    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c67p2\" (UniqueName: \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.564964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.564932    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.666424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.666394    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.666580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.666500    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.666580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.666549    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c67p2\" (UniqueName: \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.667075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.667051    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.668826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.668783    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.712803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.712718    4944 generic.go:356] "Generic (PLEG): container finished" podID="1b103112-1657-4877-b0bb-d76e9c68304a" containerID="fdbd9d49fd777ef4a8d3e90a66a99c57c6293f86cb95415bf12a1a7134ef4880" exitCode=0
Nov 03 19:08:46.713221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.712832    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6" event={"ID":"1b103112-1657-4877-b0bb-d76e9c68304a","Type":"ContainerDied","Data":"fdbd9d49fd777ef4a8d3e90a66a99c57c6293f86cb95415bf12a1a7134ef4880"}
Nov 03 19:08:46.724278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.724252    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c67p2\" (UniqueName: \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\") pod \"maas-api-revoke-sub-601b1566-s7ghx\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:46.780823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:46.780785    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:47.169215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.169179    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"]
Nov 03 19:08:47.171837 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:47.171808    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbac2237_5350_4959_8eec_71ad90cf13fc.slice/crio-c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec WatchSource:0}: Error finding container c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec: Status 404 returned error can't find the container with id c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec
Nov 03 19:08:47.665723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.665693    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:47.724921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.724820    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx" event={"ID":"dbac2237-5350-4959-8eec-71ad90cf13fc","Type":"ContainerStarted","Data":"26d8408480b3a80caed989da0bb577fe4117038d78e675355a8bcc9913ddc92a"}
Nov 03 19:08:47.725350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.724926    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx" event={"ID":"dbac2237-5350-4959-8eec-71ad90cf13fc","Type":"ContainerStarted","Data":"c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec"}
Nov 03 19:08:47.844360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.844251    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx" podStartSLOduration=1.844227056 podStartE2EDuration="1.844227056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:47.832232774 +0000 UTC m=+1426.109586359" watchObservedRunningTime="2026-11-03 19:08:47.844227056 +0000 UTC m=+1426.121580640"
Nov 03 19:08:47.903592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.903563    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:47.986467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.986390    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\" (UniqueName: \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\") pod \"1b103112-1657-4877-b0bb-d76e9c68304a\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") "
Nov 03 19:08:47.986619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.986557    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\") pod \"1b103112-1657-4877-b0bb-d76e9c68304a\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") "
Nov 03 19:08:47.986664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.986628    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\") pod \"1b103112-1657-4877-b0bb-d76e9c68304a\" (UID: \"1b103112-1657-4877-b0bb-d76e9c68304a\") "
Nov 03 19:08:47.986927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.986899    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca" pod "1b103112-1657-4877-b0bb-d76e9c68304a" (UID: "1b103112-1657-4877-b0bb-d76e9c68304a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:47.987254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.987231    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:47.988622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.988599    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca" pod "1b103112-1657-4877-b0bb-d76e9c68304a" (UID: "1b103112-1657-4877-b0bb-d76e9c68304a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:47.988693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:47.988670    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6" pod "1b103112-1657-4877-b0bb-d76e9c68304a" (UID: "1b103112-1657-4877-b0bb-d76e9c68304a"). InnerVolumeSpecName "kube-api-access-zb8l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:48.088755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.088709    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b103112-1657-4877-b0bb-d76e9c68304a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:48.088755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.088754    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8l6\" (UniqueName: \"kubernetes.io/projected/1b103112-1657-4877-b0bb-d76e9c68304a-kube-api-access-zb8l6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:48.738201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.738164    4944 generic.go:356] "Generic (PLEG): container finished" podID="dbac2237-5350-4959-8eec-71ad90cf13fc" containerID="26d8408480b3a80caed989da0bb577fe4117038d78e675355a8bcc9913ddc92a" exitCode=0
Nov 03 19:08:48.738571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.738209    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx" event={"ID":"dbac2237-5350-4959-8eec-71ad90cf13fc","Type":"ContainerDied","Data":"26d8408480b3a80caed989da0bb577fe4117038d78e675355a8bcc9913ddc92a"}
Nov 03 19:08:48.739771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.739739    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"
Nov 03 19:08:48.739771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.739758    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6" event={"ID":"1b103112-1657-4877-b0bb-d76e9c68304a","Type":"ContainerDied","Data":"0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60"}
Nov 03 19:08:48.739949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:48.739825    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0aec49214575e92ba2eca2dc696a0ad5cef7cd1c56a9c3dddf84c5b2f32daa60"
Nov 03 19:08:49.321268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.321227    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"]
Nov 03 19:08:49.321976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.321960    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1b103112-1657-4877-b0bb-d76e9c68304a" containerName="revoke-keys"
Nov 03 19:08:49.322041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.321978    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b103112-1657-4877-b0bb-d76e9c68304a" containerName="revoke-keys"
Nov 03 19:08:49.322111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.322101    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b103112-1657-4877-b0bb-d76e9c68304a" containerName="revoke-keys"
Nov 03 19:08:49.337827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.337777    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.380685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.380632    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"]
Nov 03 19:08:49.407863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.407833    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.408126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.408091    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c57\" (UniqueName: \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.408258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.408186    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.509126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.509091    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.509324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.509311    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c57\" (UniqueName: \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.509413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.509338    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.510243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.510216    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.511612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.511589    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.566269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.566241    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c57\" (UniqueName: \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\") pod \"maas-api-revoke-sub-7f1654be-tzr9s\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.650603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.650520    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:49.938669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:49.938640    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:50.016259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.016221    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\") pod \"dbac2237-5350-4959-8eec-71ad90cf13fc\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") "
Nov 03 19:08:50.016434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.016267    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\") pod \"dbac2237-5350-4959-8eec-71ad90cf13fc\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") "
Nov 03 19:08:50.016484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.016473    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\" (UniqueName: \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\") pod \"dbac2237-5350-4959-8eec-71ad90cf13fc\" (UID: \"dbac2237-5350-4959-8eec-71ad90cf13fc\") "
Nov 03 19:08:50.016681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.016658    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca" pod "dbac2237-5350-4959-8eec-71ad90cf13fc" (UID: "dbac2237-5350-4959-8eec-71ad90cf13fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:50.017233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.017208    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:50.018468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.018444    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca" pod "dbac2237-5350-4959-8eec-71ad90cf13fc" (UID: "dbac2237-5350-4959-8eec-71ad90cf13fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:50.018644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.018625    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2" pod "dbac2237-5350-4959-8eec-71ad90cf13fc" (UID: "dbac2237-5350-4959-8eec-71ad90cf13fc"). InnerVolumeSpecName "kube-api-access-c67p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:50.033227 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:50.033188    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be84a6c_e034_411a_9f48_a57865297246.slice/crio-5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2 WatchSource:0}: Error finding container 5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2: Status 404 returned error can't find the container with id 5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2
Nov 03 19:08:50.038953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.038930    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"]
Nov 03 19:08:50.118579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.118537    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbac2237-5350-4959-8eec-71ad90cf13fc-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:50.118579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.118579    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c67p2\" (UniqueName: \"kubernetes.io/projected/dbac2237-5350-4959-8eec-71ad90cf13fc-kube-api-access-c67p2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:50.760456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.760417    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s" event={"ID":"4be84a6c-e034-411a-9f48-a57865297246","Type":"ContainerStarted","Data":"171189afe9f66a17467e5801485fac0b028fa15baec30c3d2fe53572c157ca60"}
Nov 03 19:08:50.760644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.760483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s" event={"ID":"4be84a6c-e034-411a-9f48-a57865297246","Type":"ContainerStarted","Data":"5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2"}
Nov 03 19:08:50.761969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.761944    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"
Nov 03 19:08:50.762084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.761979    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx" event={"ID":"dbac2237-5350-4959-8eec-71ad90cf13fc","Type":"ContainerDied","Data":"c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec"}
Nov 03 19:08:50.762084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.762019    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c339748e464305924024443839657114c5fb8fc01fc28fe188944ae9ede063ec"
Nov 03 19:08:50.869592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:50.869539    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s" podStartSLOduration=1.8695268139999999 podStartE2EDuration="1.869526814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:50.867745028 +0000 UTC m=+1429.145098611" watchObservedRunningTime="2026-11-03 19:08:50.869526814 +0000 UTC m=+1429.146880397"
Nov 03 19:08:51.772043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:51.772013    4944 generic.go:356] "Generic (PLEG): container finished" podID="4be84a6c-e034-411a-9f48-a57865297246" containerID="171189afe9f66a17467e5801485fac0b028fa15baec30c3d2fe53572c157ca60" exitCode=0
Nov 03 19:08:51.772467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:51.772066    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s" event={"ID":"4be84a6c-e034-411a-9f48-a57865297246","Type":"ContainerDied","Data":"171189afe9f66a17467e5801485fac0b028fa15baec30c3d2fe53572c157ca60"}
Nov 03 19:08:52.051883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.051781    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"]
Nov 03 19:08:52.053309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.053145    4944 cpu_manager.go:414] "containerMap: removing container" podUID="dbac2237-5350-4959-8eec-71ad90cf13fc" containerName="revoke-keys"
Nov 03 19:08:52.053309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.053170    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbac2237-5350-4959-8eec-71ad90cf13fc" containerName="revoke-keys"
Nov 03 19:08:52.053584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.053562    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbac2237-5350-4959-8eec-71ad90cf13fc" containerName="revoke-keys"
Nov 03 19:08:52.057222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.057200    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.109717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.109683    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"]
Nov 03 19:08:52.151325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.151282    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.151494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.151333    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.151604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.151576    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whvcz\" (UniqueName: \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.253123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.253087    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whvcz\" (UniqueName: \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.253359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.253292    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.253359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.253316    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.253975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.253948    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.255953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.255925    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.301350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.301301    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:52.310474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.310411    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whvcz\" (UniqueName: \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\") pod \"maas-api-revoke-sub-1a15e442-mwbbb\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.373104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.373063    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:52.751647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.751615    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"]
Nov 03 19:08:52.755047 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:52.755015    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c33d4f0_bc77_4cd0_a20b_020b8a93cbcf.slice/crio-79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc WatchSource:0}: Error finding container 79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc: Status 404 returned error can't find the container with id 79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc
Nov 03 19:08:52.782369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.782344    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb" event={"ID":"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf","Type":"ContainerStarted","Data":"79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc"}
Nov 03 19:08:52.915317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.915293    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:52.963918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.963890    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\") pod \"4be84a6c-e034-411a-9f48-a57865297246\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") "
Nov 03 19:08:52.964111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.964037    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\" (UniqueName: \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\") pod \"4be84a6c-e034-411a-9f48-a57865297246\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") "
Nov 03 19:08:52.964191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.964163    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\") pod \"4be84a6c-e034-411a-9f48-a57865297246\" (UID: \"4be84a6c-e034-411a-9f48-a57865297246\") "
Nov 03 19:08:52.964500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.964296    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca" pod "4be84a6c-e034-411a-9f48-a57865297246" (UID: "4be84a6c-e034-411a-9f48-a57865297246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:52.965082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.965057    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4be84a6c-e034-411a-9f48-a57865297246-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:52.966397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.966368    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca" pod "4be84a6c-e034-411a-9f48-a57865297246" (UID: "4be84a6c-e034-411a-9f48-a57865297246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:52.966487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:52.966395    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57" pod "4be84a6c-e034-411a-9f48-a57865297246" (UID: "4be84a6c-e034-411a-9f48-a57865297246"). InnerVolumeSpecName "kube-api-access-n5c57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:53.066508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.066466    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5c57\" (UniqueName: \"kubernetes.io/projected/4be84a6c-e034-411a-9f48-a57865297246-kube-api-access-n5c57\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:53.066508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.066506    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4be84a6c-e034-411a-9f48-a57865297246-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:53.668328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.668295    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"]
Nov 03 19:08:53.669065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.669046    4944 cpu_manager.go:414] "containerMap: removing container" podUID="4be84a6c-e034-411a-9f48-a57865297246" containerName="revoke-keys"
Nov 03 19:08:53.669065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.669065    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4be84a6c-e034-411a-9f48-a57865297246" containerName="revoke-keys"
Nov 03 19:08:53.669225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.669165    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="4be84a6c-e034-411a-9f48-a57865297246" containerName="revoke-keys"
Nov 03 19:08:53.672046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.672027    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.725658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.725624    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"]
Nov 03 19:08:53.776103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.776075    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqk2\" (UniqueName: \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.776261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.776242    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.776308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.776285    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.792903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.792877    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"
Nov 03 19:08:53.793340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.792907    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s" event={"ID":"4be84a6c-e034-411a-9f48-a57865297246","Type":"ContainerDied","Data":"5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2"}
Nov 03 19:08:53.793340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.792952    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cc4badde07b8ba2734ac128798c2ff9ec6b47a11c97c9573f65e1f0a511b0f2"
Nov 03 19:08:53.794725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.794697    4944 generic.go:356] "Generic (PLEG): container finished" podID="6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" containerID="48ce52597929a13d893226ed39cce7f28ae450fb5b55d8639eface471879ce48" exitCode=0
Nov 03 19:08:53.794872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.794735    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb" event={"ID":"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf","Type":"ContainerDied","Data":"48ce52597929a13d893226ed39cce7f28ae450fb5b55d8639eface471879ce48"}
Nov 03 19:08:53.877911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.877878    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.878079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.877927    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.878079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.878064    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqk2\" (UniqueName: \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.878611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.878587    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.880269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.880244    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:53.939058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:53.938988    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqk2\" (UniqueName: \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\") pod \"maas-api-revoke-sub-383453ed-2qn4g\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:54.053818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:54.053764    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:54.431392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:54.431360    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"]
Nov 03 19:08:54.436320 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:54.436287    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode91fce8b_b8e5_4080_9c4a_0f21aaf7f384.slice/crio-85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e WatchSource:0}: Error finding container 85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e: Status 404 returned error can't find the container with id 85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e
Nov 03 19:08:54.803742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:54.803711    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g" event={"ID":"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384","Type":"ContainerStarted","Data":"85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e"}
Nov 03 19:08:54.984244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:54.984179    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:55.084552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.084518    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"]
Nov 03 19:08:55.089364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.089336    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09170a2-s7pvl"]
Nov 03 19:08:55.096952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.096919    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\") pod \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") "
Nov 03 19:08:55.097106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.096981    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\") pod \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") "
Nov 03 19:08:55.097182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.097145    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\" (UniqueName: \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\") pod \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\" (UID: \"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf\") "
Nov 03 19:08:55.097527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.097492    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca" pod "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" (UID: "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:55.097876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.097853    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:55.099327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.099299    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca" pod "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" (UID: "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:55.099622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.099604    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz" pod "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" (UID: "6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf"). InnerVolumeSpecName "kube-api-access-whvcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:55.198980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.198945    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:55.198980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.198975    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whvcz\" (UniqueName: \"kubernetes.io/projected/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf-kube-api-access-whvcz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:55.813277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.813244    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"
Nov 03 19:08:55.813704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.813282    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb" event={"ID":"6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf","Type":"ContainerDied","Data":"79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc"}
Nov 03 19:08:55.813704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.813359    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e12289773bb27c3177715a43168c484cbda0cd15889130b2d0f04493bf19cc"
Nov 03 19:08:55.815240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.815212    4944 generic.go:356] "Generic (PLEG): container finished" podID="e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" containerID="fe4b3849efb911f8553526a7f8d264a8c00aac73b2f8de0ae982b80488a0fa1b" exitCode=0
Nov 03 19:08:55.815346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.815283    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g" event={"ID":"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384","Type":"ContainerDied","Data":"fe4b3849efb911f8553526a7f8d264a8c00aac73b2f8de0ae982b80488a0fa1b"}
Nov 03 19:08:55.939914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:55.939868    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951f83b9-5c64-4fdd-aa6c-aa9fd53da75b" path="/var/lib/kubelet/pods/951f83b9-5c64-4fdd-aa6c-aa9fd53da75b/volumes"
Nov 03 19:08:56.047102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.047070    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"]
Nov 03 19:08:56.051752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.051727    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25c8719d-mhltc"]
Nov 03 19:08:56.730211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.730107    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"]
Nov 03 19:08:56.753989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.730783    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" containerName="revoke-keys"
Nov 03 19:08:56.753989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.730809    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" containerName="revoke-keys"
Nov 03 19:08:56.753989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.730934    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" containerName="revoke-keys"
Nov 03 19:08:56.753989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.733989    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.788986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.788953    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"]
Nov 03 19:08:56.825133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.823734    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.825133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.823817    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.825133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.823897    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgqj\" (UniqueName: \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.831043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.831006    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:08:56.925356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.925316    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.925544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.925365    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.925544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.925434    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgqj\" (UniqueName: \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.933702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.931522    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.935588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.935541    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:56.983745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:56.983677    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgqj\" (UniqueName: \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\") pod \"maas-api-revoke-sub-7cb64e58-rlbdf\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:57.007913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.007889    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:57.047844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.047811    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:08:57.127804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.127753    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\") pod \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") "
Nov 03 19:08:57.128007 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.127985    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\" (UniqueName: \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\") pod \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") "
Nov 03 19:08:57.128080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.128052    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\") pod \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\" (UID: \"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384\") "
Nov 03 19:08:57.131333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.131300    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca" pod "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" (UID: "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:08:57.132068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.132024    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca" pod "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" (UID: "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:08:57.140222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.140189    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"]
Nov 03 19:08:57.141220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.141191    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2" pod "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" (UID: "e91fce8b-b8e5-4080-9c4a-0f21aaf7f384"). InnerVolumeSpecName "kube-api-access-wnqk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:08:57.141503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.141479    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" containerName="revoke-keys"
Nov 03 19:08:57.141567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.141508    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" containerName="revoke-keys"
Nov 03 19:08:57.141738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.141722    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" containerName="revoke-keys"
Nov 03 19:08:57.146371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.146349    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.198915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.198876    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"]
Nov 03 19:08:57.230003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.229970    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfnb\" (UniqueName: \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.230275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.230245    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.230326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.230309    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.230516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.230503    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnqk2\" (UniqueName: \"kubernetes.io/projected/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-kube-api-access-wnqk2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:57.230555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.230523    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:57.230555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.230539    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:08:57.332171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.332138    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.332365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.332188    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.332365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.332315    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfnb\" (UniqueName: \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.332930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.332905    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.334474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.334455    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.389163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.389131    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfnb\" (UniqueName: \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\") pod \"maas-api-revoke-sub-6ac9570e-bqp7g\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.425776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.425749    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"]
Nov 03 19:08:57.428805 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:57.428753    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aee447_e413_41fe_9bac_139641ff7aa8.slice/crio-620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0 WatchSource:0}: Error finding container 620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0: Status 404 returned error can't find the container with id 620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0
Nov 03 19:08:57.464983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.464959    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:08:57.840703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.840659    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf" event={"ID":"58aee447-e413-41fe-9bac-139641ff7aa8","Type":"ContainerStarted","Data":"27e486c63fde249103db4e54e6e496b109dd37a66766294bda882e4e6d7438c4"}
Nov 03 19:08:57.841185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.840776    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf" event={"ID":"58aee447-e413-41fe-9bac-139641ff7aa8","Type":"ContainerStarted","Data":"620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0"}
Nov 03 19:08:57.842581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.842558    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g" event={"ID":"e91fce8b-b8e5-4080-9c4a-0f21aaf7f384","Type":"ContainerDied","Data":"85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e"}
Nov 03 19:08:57.842581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.842580    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"
Nov 03 19:08:57.842765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.842600    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85653ccf999cb7a76cd7f33efbd921af00c366fbb766802e05fcae68ea7cf34e"
Nov 03 19:08:57.846290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.846268    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"]
Nov 03 19:08:57.848585 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:08:57.848554    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798e000f_5bf6_44c0_9750_8ec25e46b3fa.slice/crio-313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20 WatchSource:0}: Error finding container 313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20: Status 404 returned error can't find the container with id 313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20
Nov 03 19:08:57.936187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.936146    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0066318e-55be-486e-a19a-7791a7fe58ea" path="/var/lib/kubelet/pods/0066318e-55be-486e-a19a-7791a7fe58ea/volumes"
Nov 03 19:08:57.949407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:57.949354    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf" podStartSLOduration=1.949338833 podStartE2EDuration="1.949338833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:08:57.947925126 +0000 UTC m=+1436.225278710" watchObservedRunningTime="2026-11-03 19:08:57.949338833 +0000 UTC m=+1436.226692410"
Nov 03 19:08:58.047229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.047191    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"]
Nov 03 19:08:58.051963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.051935    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4df9a87-6c2ks"]
Nov 03 19:08:58.852742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.852701    4944 generic.go:356] "Generic (PLEG): container finished" podID="58aee447-e413-41fe-9bac-139641ff7aa8" containerID="27e486c63fde249103db4e54e6e496b109dd37a66766294bda882e4e6d7438c4" exitCode=0
Nov 03 19:08:58.853179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.852781    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf" event={"ID":"58aee447-e413-41fe-9bac-139641ff7aa8","Type":"ContainerDied","Data":"27e486c63fde249103db4e54e6e496b109dd37a66766294bda882e4e6d7438c4"}
Nov 03 19:08:58.854471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.854446    4944 generic.go:356] "Generic (PLEG): container finished" podID="798e000f-5bf6-44c0-9750-8ec25e46b3fa" containerID="cff864f33a868f3b24c32399803c0721e6cf27a4f022187f1775d3c964b1ebb9" exitCode=0
Nov 03 19:08:58.854599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.854520    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g" event={"ID":"798e000f-5bf6-44c0-9750-8ec25e46b3fa","Type":"ContainerDied","Data":"cff864f33a868f3b24c32399803c0721e6cf27a4f022187f1775d3c964b1ebb9"}
Nov 03 19:08:58.854599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:58.854557    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g" event={"ID":"798e000f-5bf6-44c0-9750-8ec25e46b3fa","Type":"ContainerStarted","Data":"313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20"}
Nov 03 19:08:59.945938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:59.945900    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3636feb3-201f-454b-992e-c82ef014d067" path="/var/lib/kubelet/pods/3636feb3-201f-454b-992e-c82ef014d067/volumes"
Nov 03 19:08:59.994832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:08:59.994800    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"]
Nov 03 19:09:00.000321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.000106    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.041502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.041468    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:09:00.056775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.052227    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"]
Nov 03 19:09:00.106127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.106104    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:09:00.172768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.172691    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\" (UniqueName: \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\") pod \"58aee447-e413-41fe-9bac-139641ff7aa8\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") "
Nov 03 19:09:00.173039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173012    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\") pod \"58aee447-e413-41fe-9bac-139641ff7aa8\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") "
Nov 03 19:09:00.173163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173055    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\") pod \"58aee447-e413-41fe-9bac-139641ff7aa8\" (UID: \"58aee447-e413-41fe-9bac-139641ff7aa8\") "
Nov 03 19:09:00.173374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.173509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173483    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca" pod "58aee447-e413-41fe-9bac-139641ff7aa8" (UID: "58aee447-e413-41fe-9bac-139641ff7aa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:00.173727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173701    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.173883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173758    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57n4t\" (UniqueName: \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.173954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.173898    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.175062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.175029    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj" pod "58aee447-e413-41fe-9bac-139641ff7aa8" (UID: "58aee447-e413-41fe-9bac-139641ff7aa8"). InnerVolumeSpecName "kube-api-access-bvgqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:00.175143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.175096    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca" pod "58aee447-e413-41fe-9bac-139641ff7aa8" (UID: "58aee447-e413-41fe-9bac-139641ff7aa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:00.182974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.182954    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:09:00.183752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183734    4944 cpu_manager.go:414] "containerMap: removing container" podUID="58aee447-e413-41fe-9bac-139641ff7aa8" containerName="revoke-keys"
Nov 03 19:09:00.183837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183753    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58aee447-e413-41fe-9bac-139641ff7aa8" containerName="revoke-keys"
Nov 03 19:09:00.183837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183764    4944 cpu_manager.go:414] "containerMap: removing container" podUID="798e000f-5bf6-44c0-9750-8ec25e46b3fa" containerName="revoke-keys"
Nov 03 19:09:00.183837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183769    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798e000f-5bf6-44c0-9750-8ec25e46b3fa" containerName="revoke-keys"
Nov 03 19:09:00.183945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183906    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="58aee447-e413-41fe-9bac-139641ff7aa8" containerName="revoke-keys"
Nov 03 19:09:00.183945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.183918    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="798e000f-5bf6-44c0-9750-8ec25e46b3fa" containerName="revoke-keys"
Nov 03 19:09:00.186992 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.186977    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.274752    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\" (UniqueName: \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\") pod \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") "
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275123    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\") pod \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") "
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275185    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\") pod \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\" (UID: \"798e000f-5bf6-44c0-9750-8ec25e46b3fa\") "
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275479    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275515    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275582    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275620    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57n4t\" (UniqueName: \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275704    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wd6r\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275739    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275740    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.275775    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.276169    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca" pod "798e000f-5bf6-44c0-9750-8ec25e46b3fa" (UID: "798e000f-5bf6-44c0-9750-8ec25e46b3fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277053    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277143    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277173    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277251    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277316    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277458    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58aee447-e413-41fe-9bac-139641ff7aa8-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277476    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.277491    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgqj\" (UniqueName: \"kubernetes.io/projected/58aee447-e413-41fe-9bac-139641ff7aa8-kube-api-access-bvgqj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.278110    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.279155    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.279881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.279764    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb" pod "798e000f-5bf6-44c0-9750-8ec25e46b3fa" (UID: "798e000f-5bf6-44c0-9750-8ec25e46b3fa"). InnerVolumeSpecName "kube-api-access-zmfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:00.281582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.281554    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca" pod "798e000f-5bf6-44c0-9750-8ec25e46b3fa" (UID: "798e000f-5bf6-44c0-9750-8ec25e46b3fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:00.334297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.334264    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57n4t\" (UniqueName: \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\") pod \"maas-api-revoke-sub-c8d6f1cc-ggjb5\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.379224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.379195    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.379396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.379249    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.379396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.379329    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.379635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.379612    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.379936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.379910    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wd6r\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380077    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380015    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380249    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380367    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380445    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380479    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380547    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380600    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798e000f-5bf6-44c0-9750-8ec25e46b3fa-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380620    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfnb\" (UniqueName: \"kubernetes.io/projected/798e000f-5bf6-44c0-9750-8ec25e46b3fa-kube-api-access-zmfnb\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:00.380844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380819    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.381211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.380885    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.388705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.388677    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.388935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.388912    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.405784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.405759    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:00.435980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.435910    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.436399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.436377    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wd6r\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\") pod \"e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") " pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.500143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.500109    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:00.787004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.786978    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"]
Nov 03 19:09:00.789805 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:00.789769    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c29b35e_acdc_43a8_b34a_e623eb279238.slice/crio-208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546 WatchSource:0}: Error finding container 208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546: Status 404 returned error can't find the container with id 208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546
Nov 03 19:09:00.872531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.872502    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"
Nov 03 19:09:00.872700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.872525    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf" event={"ID":"58aee447-e413-41fe-9bac-139641ff7aa8","Type":"ContainerDied","Data":"620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0"}
Nov 03 19:09:00.872700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.872576    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="620242ef9057494dc8210fa8e0f69433e3aca0ec6a175a6eda6463fdb367dea0"
Nov 03 19:09:00.874017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.873996    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"
Nov 03 19:09:00.874146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.874040    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g" event={"ID":"798e000f-5bf6-44c0-9750-8ec25e46b3fa","Type":"ContainerDied","Data":"313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20"}
Nov 03 19:09:00.874146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.874080    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="313455c1240ec7da495118faa35306e31eb701175babcf09590fd6c7ecf1ad20"
Nov 03 19:09:00.875536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.875507    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5" event={"ID":"5c29b35e-acdc-43a8-b34a-e623eb279238","Type":"ContainerStarted","Data":"208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546"}
Nov 03 19:09:00.893706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.893684    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:09:00.895836 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:00.895783    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1f81f7_c4f0_469f_87e8_ccda6a51ee3b.slice/crio-3a603bdbd01e39e206d7138e72ab7b51307898e1170e4e485d1b34f2911b782a WatchSource:0}: Error finding container 3a603bdbd01e39e206d7138e72ab7b51307898e1170e4e485d1b34f2911b782a: Status 404 returned error can't find the container with id 3a603bdbd01e39e206d7138e72ab7b51307898e1170e4e485d1b34f2911b782a
Nov 03 19:09:00.898329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.898300    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:09:00.898408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.898361    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:09:00.898408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:00.898390    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:09:01.461094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:01.460994    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:01.885189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:01.885156    4944 generic.go:356] "Generic (PLEG): container finished" podID="5c29b35e-acdc-43a8-b34a-e623eb279238" containerID="d7403fc90ab63e95604053d504554c1d2c82a4038fb6ea90c888e4cd220a2d33" exitCode=0
Nov 03 19:09:01.885374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:01.885246    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5" event={"ID":"5c29b35e-acdc-43a8-b34a-e623eb279238","Type":"ContainerDied","Data":"d7403fc90ab63e95604053d504554c1d2c82a4038fb6ea90c888e4cd220a2d33"}
Nov 03 19:09:01.886846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:01.886819    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" event={"ID":"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b","Type":"ContainerStarted","Data":"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"}
Nov 03 19:09:01.886954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:01.886867    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" event={"ID":"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b","Type":"ContainerStarted","Data":"3a603bdbd01e39e206d7138e72ab7b51307898e1170e4e485d1b34f2911b782a"}
Nov 03 19:09:02.135187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.135038    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podStartSLOduration=2.135019069 podStartE2EDuration="2.135019069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:02.12721976 +0000 UTC m=+1440.404573367" watchObservedRunningTime="2026-11-03 19:09:02.135019069 +0000 UTC m=+1440.412372653"
Nov 03 19:09:02.201556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.201523    4944 scope.go:122] "RemoveContainer" containerID="87c0beccab60542d685d3d440aa0940b1f02e60f9e53009b1ac0503ed7c99dad"
Nov 03 19:09:02.242549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.242522    4944 scope.go:122] "RemoveContainer" containerID="34a3e8463514b07d825e22bebef56ead810c6f3821e9ca9d48ca4b70975aeff9"
Nov 03 19:09:02.253722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.253700    4944 scope.go:122] "RemoveContainer" containerID="a5786b3a06533d5f824c4dbb2d4fb7ac17a3ce1d3bc0a99ee6a856106caf6238"
Nov 03 19:09:02.263546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.263521    4944 scope.go:122] "RemoveContainer" containerID="0aa3a4d023250f314ef2deece4619f7e54a6d11afa289aa9424f0041fd0c1f4b"
Nov 03 19:09:02.272799 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.272773    4944 scope.go:122] "RemoveContainer" containerID="0ea1525706ea3330b7c1c7a1b6971c40cc5e971bdfeaefe96d5d6e481c7d49a3"
Nov 03 19:09:02.284439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.284419    4944 scope.go:122] "RemoveContainer" containerID="0a7315df14ca7dfa9d2110989ee2fe979a9bbad5f5bfcb31f30865756fd6ead2"
Nov 03 19:09:02.293955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.293935    4944 scope.go:122] "RemoveContainer" containerID="8c6556c0cda17a27e701573df84b8c56ed79b40a41eb5689f7a21166f3463a44"
Nov 03 19:09:02.302857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.302835    4944 scope.go:122] "RemoveContainer" containerID="e162295ac7b14ac2aac04d98ba5d56f6606f1bf661f756fddb3ecdecc8521d0d"
Nov 03 19:09:02.334447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.334418    4944 scope.go:122] "RemoveContainer" containerID="0d58303ec80f43cf7fcc28100f4d111fbf9d90e1965178f042721c7beee0045e"
Nov 03 19:09:02.345020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.344979    4944 scope.go:122] "RemoveContainer" containerID="06c9def470bdcba9b3c41ffb53b737a8858f49b3638cfbc36e4704aef0e59b3a"
Nov 03 19:09:02.501294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.501270    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:02.502942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.502920    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:02.503036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:02.502978    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:03.134577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.134546    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:03.255730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.255699    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\" (UniqueName: \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\") pod \"5c29b35e-acdc-43a8-b34a-e623eb279238\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") "
Nov 03 19:09:03.255967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.255838    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\") pod \"5c29b35e-acdc-43a8-b34a-e623eb279238\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") "
Nov 03 19:09:03.255967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.255920    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\") pod \"5c29b35e-acdc-43a8-b34a-e623eb279238\" (UID: \"5c29b35e-acdc-43a8-b34a-e623eb279238\") "
Nov 03 19:09:03.256247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.256218    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca" pod "5c29b35e-acdc-43a8-b34a-e623eb279238" (UID: "5c29b35e-acdc-43a8-b34a-e623eb279238"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:03.256739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.256687    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:03.258905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.258877    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t" pod "5c29b35e-acdc-43a8-b34a-e623eb279238" (UID: "5c29b35e-acdc-43a8-b34a-e623eb279238"). InnerVolumeSpecName "kube-api-access-57n4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:03.259050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.259027    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca" pod "5c29b35e-acdc-43a8-b34a-e623eb279238" (UID: "5c29b35e-acdc-43a8-b34a-e623eb279238"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:03.358563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.358518    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57n4t\" (UniqueName: \"kubernetes.io/projected/5c29b35e-acdc-43a8-b34a-e623eb279238-kube-api-access-57n4t\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:03.358563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.358559    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c29b35e-acdc-43a8-b34a-e623eb279238-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:03.500933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.500895    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:03.501107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.500955    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:03.904613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.904582    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"
Nov 03 19:09:03.905023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.904653    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5" event={"ID":"5c29b35e-acdc-43a8-b34a-e623eb279238","Type":"ContainerDied","Data":"208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546"}
Nov 03 19:09:03.905023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:03.904703    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="208fa7483ae8f9fb618ca9ea32469eae14705b3a46c0c8104ff0da141ad1a546"
Nov 03 19:09:04.501062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:04.501024    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:04.501264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:04.501101    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:05.247757 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:09:05.247722    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:09:05.500863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:05.500759    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:05.500863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:05.500836    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:06.500605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:06.500570    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:06.501026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:06.500625    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:06.576332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:06.576276    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:07.501099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:07.501062    4944 patch_prober.go:29] interesting pod/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Nov 03 19:09:07.501507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:07.501113    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Nov 03 19:09:07.914892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:07.914800    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:08.067457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.067420    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"]
Nov 03 19:09:08.116678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.116641    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a76c8f89-phl4j"]
Nov 03 19:09:08.233441 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.233351    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:08.505609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.505566    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:08.506077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.506053    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:08.506650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.506631    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:09:08.571448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.571419    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:08.829186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:08.829107    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:09.175269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.175191    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:09.426089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.425996    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:09.679606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.679496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:09.880167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.880134    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"]
Nov 03 19:09:09.880943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.880925    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5c29b35e-acdc-43a8-b34a-e623eb279238" containerName="revoke-keys"
Nov 03 19:09:09.880993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.880945    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c29b35e-acdc-43a8-b34a-e623eb279238" containerName="revoke-keys"
Nov 03 19:09:09.881062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.881054    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c29b35e-acdc-43a8-b34a-e623eb279238" containerName="revoke-keys"
Nov 03 19:09:09.884332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.884305    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:09.933385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.933309    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:09:09.950693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.947877    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="347b2196-41c2-48e6-ba70-cc93dbe94cde" path="/var/lib/kubelet/pods/347b2196-41c2-48e6-ba70-cc93dbe94cde/volumes"
Nov 03 19:09:09.950693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.948055    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:09.950693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.948588    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"]
Nov 03 19:09:09.977864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.977836    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:09:09.987439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.987407    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:09:09.987696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:09.987500    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:09:10.014067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.014022    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.014258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.014231    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9dbn\" (UniqueName: \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.014413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.014393    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.048647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.048614    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"]
Nov 03 19:09:10.058520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.058492    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23c7149f-lpxhr"]
Nov 03 19:09:10.116032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.115995    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9dbn\" (UniqueName: \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.116228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.116202    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.116361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.116338    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.116923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.116900    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.118512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.118485    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.173391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.173363    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9dbn\" (UniqueName: \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\") pod \"maas-api-revoke-sub-0e2b203a-kwnjw\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.200027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.199971    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:10.578860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.578836    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"]
Nov 03 19:09:10.581988 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:10.581957    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698a7136_f5f3_4d97_944f_0263aa79be28.slice/crio-660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e WatchSource:0}: Error finding container 660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e: Status 404 returned error can't find the container with id 660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e
Nov 03 19:09:10.996039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.996006    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw" event={"ID":"698a7136-f5f3-4d97-944f-0263aa79be28","Type":"ContainerStarted","Data":"296b5cc513e1d786eeccc1f21a0c8fb4191e34310d857566e469921db90f9536"}
Nov 03 19:09:10.996513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:10.996067    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw" event={"ID":"698a7136-f5f3-4d97-944f-0263aa79be28","Type":"ContainerStarted","Data":"660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e"}
Nov 03 19:09:11.050726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.050689    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"]
Nov 03 19:09:11.055431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.055406    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f7c2455-2x8f8"]
Nov 03 19:09:11.108764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.108665    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw" podStartSLOduration=2.108650938 podStartE2EDuration="2.108650938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:11.103812852 +0000 UTC m=+1449.381166429" watchObservedRunningTime="2026-11-03 19:09:11.108650938 +0000 UTC m=+1449.386004569"
Nov 03 19:09:11.538215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.538177    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:11.602867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.602834    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"]
Nov 03 19:09:11.608952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.608922    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.663404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.662918    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"]
Nov 03 19:09:11.743125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.743091    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.743295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.743179    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsngx\" (UniqueName: \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.743295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.743260    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.785403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.785323    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:11.844391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.844361    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.844518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.844399    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsngx\" (UniqueName: \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.844518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.844451    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.845173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.845153    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.846815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.846759    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.902570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.902534    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsngx\" (UniqueName: \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\") pod \"maas-api-revoke-sub-62723de0-7clnb\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.925939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.925909    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:11.937868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.937838    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5697f688-855e-4b38-8639-7f1b840c352d" path="/var/lib/kubelet/pods/5697f688-855e-4b38-8639-7f1b840c352d/volumes"
Nov 03 19:09:11.938522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:11.938498    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5ca783-df83-4149-946a-78b670d1f058" path="/var/lib/kubelet/pods/8e5ca783-df83-4149-946a-78b670d1f058/volumes"
Nov 03 19:09:12.035687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:12.035616    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:12.037271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:12.037244    4944 generic.go:356] "Generic (PLEG): container finished" podID="698a7136-f5f3-4d97-944f-0263aa79be28" containerID="296b5cc513e1d786eeccc1f21a0c8fb4191e34310d857566e469921db90f9536" exitCode=0
Nov 03 19:09:12.037383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:12.037323    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw" event={"ID":"698a7136-f5f3-4d97-944f-0263aa79be28","Type":"ContainerDied","Data":"296b5cc513e1d786eeccc1f21a0c8fb4191e34310d857566e469921db90f9536"}
Nov 03 19:09:12.324209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:12.324134    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"]
Nov 03 19:09:12.327579 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:12.327550    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7598aedf_d8a8_4122_b232_cbc9f5a1018d.slice/crio-d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005 WatchSource:0}: Error finding container d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005: Status 404 returned error can't find the container with id d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005
Nov 03 19:09:12.539721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:12.539686    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:13.047212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.047179    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb" event={"ID":"7598aedf-d8a8-4122-b232-cbc9f5a1018d","Type":"ContainerStarted","Data":"f271c108125d6dd4857b604ad48474030cbf347c861ee6a4f3c5214ea8aee3b1"}
Nov 03 19:09:13.047707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.047234    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb" event={"ID":"7598aedf-d8a8-4122-b232-cbc9f5a1018d","Type":"ContainerStarted","Data":"d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005"}
Nov 03 19:09:13.125706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.124529    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:13.161249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.161190    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb" podStartSLOduration=2.161175816 podStartE2EDuration="2.161175816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:13.157537136 +0000 UTC m=+1451.434890720" watchObservedRunningTime="2026-11-03 19:09:13.161175816 +0000 UTC m=+1451.438529441"
Nov 03 19:09:13.227605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.227577    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:13.295812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.295333    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\") pod \"698a7136-f5f3-4d97-944f-0263aa79be28\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") "
Nov 03 19:09:13.295812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.295467    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\") pod \"698a7136-f5f3-4d97-944f-0263aa79be28\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") "
Nov 03 19:09:13.295812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.295734    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\" (UniqueName: \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\") pod \"698a7136-f5f3-4d97-944f-0263aa79be28\" (UID: \"698a7136-f5f3-4d97-944f-0263aa79be28\") "
Nov 03 19:09:13.325861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.305995    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca" pod "698a7136-f5f3-4d97-944f-0263aa79be28" (UID: "698a7136-f5f3-4d97-944f-0263aa79be28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:13.327526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.327483    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn" pod "698a7136-f5f3-4d97-944f-0263aa79be28" (UID: "698a7136-f5f3-4d97-944f-0263aa79be28"). InnerVolumeSpecName "kube-api-access-d9dbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:13.352818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.348720    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca" pod "698a7136-f5f3-4d97-944f-0263aa79be28" (UID: "698a7136-f5f3-4d97-944f-0263aa79be28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:13.412560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.411481    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:13.421535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.421497    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:13.421535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.421531    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9dbn\" (UniqueName: \"kubernetes.io/projected/698a7136-f5f3-4d97-944f-0263aa79be28-kube-api-access-d9dbn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:13.421535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:13.421543    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/698a7136-f5f3-4d97-944f-0263aa79be28-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:14.058547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.058509    4944 generic.go:356] "Generic (PLEG): container finished" podID="7598aedf-d8a8-4122-b232-cbc9f5a1018d" containerID="f271c108125d6dd4857b604ad48474030cbf347c861ee6a4f3c5214ea8aee3b1" exitCode=0
Nov 03 19:09:14.058977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.058592    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb" event={"ID":"7598aedf-d8a8-4122-b232-cbc9f5a1018d","Type":"ContainerDied","Data":"f271c108125d6dd4857b604ad48474030cbf347c861ee6a4f3c5214ea8aee3b1"}
Nov 03 19:09:14.060302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.060269    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"
Nov 03 19:09:14.060302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.060293    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw" event={"ID":"698a7136-f5f3-4d97-944f-0263aa79be28","Type":"ContainerDied","Data":"660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e"}
Nov 03 19:09:14.060509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.060341    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="660e708b465a86208b1bb11dfa5aff5761a7a465ee8b5cc863bc07f0ab41529e"
Nov 03 19:09:14.784627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:14.784595    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:15.036406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.036323    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:15.086666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.086631    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"]
Nov 03 19:09:15.087550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.087526    4944 cpu_manager.go:414] "containerMap: removing container" podUID="698a7136-f5f3-4d97-944f-0263aa79be28" containerName="revoke-keys"
Nov 03 19:09:15.087651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.087551    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="698a7136-f5f3-4d97-944f-0263aa79be28" containerName="revoke-keys"
Nov 03 19:09:15.087782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.087769    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="698a7136-f5f3-4d97-944f-0263aa79be28" containerName="revoke-keys"
Nov 03 19:09:15.090890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.090868    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.180365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.180317    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.180903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.180879    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjswf\" (UniqueName: \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.181126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.181110    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.275094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.275056    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"]
Nov 03 19:09:15.325104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.310288    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.325104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.310427    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.325104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.310635    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjswf\" (UniqueName: \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.325104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.311816    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.325104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.312690    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:15.337380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.337265    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.377825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.377777    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjswf\" (UniqueName: \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\") pod \"maas-api-revoke-sub-904c41b6-n857f\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.407130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.407104    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:15.407362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.407227    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:15.513467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.513434    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\") pod \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") "
Nov 03 19:09:15.513844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.513508    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\" (UniqueName: \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\") pod \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") "
Nov 03 19:09:15.513844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.513584    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\") pod \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\" (UID: \"7598aedf-d8a8-4122-b232-cbc9f5a1018d\") "
Nov 03 19:09:15.514767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.514736    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca" pod "7598aedf-d8a8-4122-b232-cbc9f5a1018d" (UID: "7598aedf-d8a8-4122-b232-cbc9f5a1018d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:15.516500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.516285    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx" pod "7598aedf-d8a8-4122-b232-cbc9f5a1018d" (UID: "7598aedf-d8a8-4122-b232-cbc9f5a1018d"). InnerVolumeSpecName "kube-api-access-xsngx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:15.516946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.516920    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca" pod "7598aedf-d8a8-4122-b232-cbc9f5a1018d" (UID: "7598aedf-d8a8-4122-b232-cbc9f5a1018d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:15.631559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.631003    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:15.631559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.631041    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7598aedf-d8a8-4122-b232-cbc9f5a1018d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:15.631559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.631056    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsngx\" (UniqueName: \"kubernetes.io/projected/7598aedf-d8a8-4122-b232-cbc9f5a1018d-kube-api-access-xsngx\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:15.958247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:15.957526    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"]
Nov 03 19:09:15.961139 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:15.961110    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee201c8_5c45_46ba_b56d_b996d45fcab4.slice/crio-2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782 WatchSource:0}: Error finding container 2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782: Status 404 returned error can't find the container with id 2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782
Nov 03 19:09:16.214900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:16.214781    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f" event={"ID":"5ee201c8-5c45-46ba-b56d-b996d45fcab4","Type":"ContainerStarted","Data":"2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782"}
Nov 03 19:09:16.227874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:16.227835    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb" event={"ID":"7598aedf-d8a8-4122-b232-cbc9f5a1018d","Type":"ContainerDied","Data":"d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005"}
Nov 03 19:09:16.228137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:16.228117    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2c53d74f732c572529a0a8043e53d44479a4e3ed2e0570a2a392a7fef054005"
Nov 03 19:09:16.228310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:16.228297    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"
Nov 03 19:09:16.823828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:16.800514    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:17.118861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.118829    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:17.273510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.272888    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.417392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.416111    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.475826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.461877    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.475826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.468200    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f" event={"ID":"5ee201c8-5c45-46ba-b56d-b996d45fcab4","Type":"ContainerStarted","Data":"db3cb703ba34c8fc73cd9d8bffcbdc0959138b0d95f210b5d6298b041bb09732"}
Nov 03 19:09:17.476139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.476115    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:09:17.487813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.477346    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7598aedf-d8a8-4122-b232-cbc9f5a1018d" containerName="revoke-keys"
Nov 03 19:09:17.488041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.488019    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7598aedf-d8a8-4122-b232-cbc9f5a1018d" containerName="revoke-keys"
Nov 03 19:09:17.488400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.488383    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7598aedf-d8a8-4122-b232-cbc9f5a1018d" containerName="revoke-keys"
Nov 03 19:09:17.517014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.503183    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.517014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.511073    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.522957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.522921    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.611815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.553299    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.613234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.571835    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.613482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.613458    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p956n\" (UniqueName: \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.614303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.573513    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.615186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.601128    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-46ccd045-dockercfg-bkb52\""
Nov 03 19:09:17.615328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.600531    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-46ccd045\""
Nov 03 19:09:17.713403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.713356    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.747910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.721810    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.748218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.748190    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p956n\" (UniqueName: \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.775708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.722686    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.854893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.854851    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:09:17.868811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.860577    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p956n\" (UniqueName: \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\") pod \"payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") " pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:17.922865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.919125    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:09:17.928873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.928843    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:17.954821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.954300    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.956538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.955030    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:09:17.956538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.955420    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:17.985301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.985267    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:17.985845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:17.985817    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtccn\" (UniqueName: \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.003865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.001120    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:09:18.019446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.017071    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:18.127970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.125516    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.159936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.139201    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f" podStartSLOduration=3.139178388 podStartE2EDuration="3.139178388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:17.601310785 +0000 UTC m=+1455.878664392" watchObservedRunningTime="2026-11-03 19:09:18.139178388 +0000 UTC m=+1456.416531972"
Nov 03 19:09:18.284911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.218174    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtccn\" (UniqueName: \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.284911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.218504    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.286431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.286399    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.407816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.307779    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.474816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.313477    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:18.474816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.356139    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.474816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.356490    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:09:18.501103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.485145    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtccn\" (UniqueName: \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\") pod \"payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.517900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.487697    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:09:18.546868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.545002    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:09:18.601814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.599108    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:09:18.633815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.608742    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:09:18.737812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.715102    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.737812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.737717    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.738574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.738538    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:18.758869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.742115    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:09:18.761255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.761213    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:09:18.762973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.762955    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:09:18.763209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.763171    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:09:18.799988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.790296    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:18.941815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.898673    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:09:18.941815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.899105    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:09:19.003814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.976451    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.003814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.976560    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.003814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.977521    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:09:19.003814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:18.977584    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:09:19.100045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.099942    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:19.149653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.149605    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.286156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.286125    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:09:19.392387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.392291    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:19.498496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.498431    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:09:19.502812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.499151    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:09:19.502812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.499479    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:09:19.502812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.499931    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.622549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.598457    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:09:19.649140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.649024    4944 generic.go:356] "Generic (PLEG): container finished" podID="5ee201c8-5c45-46ba-b56d-b996d45fcab4" containerID="db3cb703ba34c8fc73cd9d8bffcbdc0959138b0d95f210b5d6298b041bb09732" exitCode=0
Nov 03 19:09:19.669941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.669880    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:09:19.674840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.671266    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f" event={"ID":"5ee201c8-5c45-46ba-b56d-b996d45fcab4","Type":"ContainerDied","Data":"db3cb703ba34c8fc73cd9d8bffcbdc0959138b0d95f210b5d6298b041bb09732"}
Nov 03 19:09:19.686047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.686020    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:09:19.941884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.933914    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.975815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.943584    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:09:19.977197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.977152    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:19.982812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:19.979437    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.021702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.021655    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:09:20.024623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.024574    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/health\": context deadline exceeded"
Nov 03 19:09:20.024928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.024889    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:09:20.551927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.551876    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.640833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.633246    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.614933    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.641633    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.615002    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.641984    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.607374    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:09:20.669223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.642341    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.738856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.685227    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.850433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.850328    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:09:20.850705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.850661    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.961029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.955395    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:20.961029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:20.956962    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.138285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.102451    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.138285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.121091    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.138285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.123207    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.163387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.161890    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-tx2vn" podUID="144204e6-4a23-4dfe-9686-e6da6c450145" containerName="server" probeResult="failure" output="Get \"https://10.134.0.19:8443/readyz\": context deadline exceeded"
Nov 03 19:09:21.163387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.162004    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.396337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.345950    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:09:21.396337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.346005    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.459604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.448897    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:09:21.459604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.448967    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.459604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.450705    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr" podUID="0b0bd44e-1b16-4195-9d46-c3a59e9a5534" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.641996    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.642160    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.642947    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.646505    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.647852    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:09:21.648535    4944 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.717s"
Nov 03 19:09:21.690873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.651850    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:09:21.768051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.694592    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.865879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.787340    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": context deadline exceeded"
Nov 03 19:09:21.865879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.864950    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/health\": context deadline exceeded"
Nov 03 19:09:21.956461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.956356    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:21.977853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:21.958283    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.153901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.126233    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:09:22.153901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.126581    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:09:22.213378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.154294    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.213378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.170133    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.213378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.194954    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.38:50052\" within 1s: context deadline exceeded"
Nov 03 19:09:22.214563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.214507    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.39:50052\" within 1s: context deadline exceeded"
Nov 03 19:09:22.244224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.217173    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.373160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.294675    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.381812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.378869    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:22.526013 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:22.496983    4944 logging.go:55] [core] [Channel #2556 SubChannel #2557]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.39:50052", ServerName: "10.134.0.39:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.39:50052: operation was canceled"
Nov 03 19:09:22.697815 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:22.660556    4944 logging.go:55] [core] [Channel #2552 SubChannel #2554]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.38:50052", ServerName: "10.134.0.38:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.38:50052: operation was canceled"
Nov 03 19:09:22.903870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.780849    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.092763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.905665    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.099637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.093020    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.104354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.104313    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.106335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.906181    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.107487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.014962    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.126698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.126425    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:23.172990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.172024    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.172990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.172398    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:09:23.172990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.172644    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.318078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:22.905937    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.425515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.425413    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Nov 03 19:09:23.600748 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:09:23.600715    4944 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.67s"
Nov 03 19:09:23.722279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.719037    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" event={"ID":"d9c071a1-af27-4af7-b7cb-1eb434e07caf","Type":"ContainerStarted","Data":"2a225f44d30413798f7292cecc14cad34fdb4c263de773ac68e9655ce81ddefc"}
Nov 03 19:09:23.722279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.719562    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:23.730545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.727297    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded"
Nov 03 19:09:23.734667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.734090    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:09:23.757548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.751615    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:09:23.818511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.818481    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:09:23.823446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.823420    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:09:23.824779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.824758    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:09:23.828047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.828027    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.832657    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.830843    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.834543    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.835576    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.835618    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:09:23.856844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.845690    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:09:23.860246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.860224    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:09:23.873735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.873603    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:09:23.885623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.885599    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:09:23.890946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.890922    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:09:23.895804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.895774    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"]
Nov 03 19:09:23.896576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.896559    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5ee201c8-5c45-46ba-b56d-b996d45fcab4" containerName="revoke-keys"
Nov 03 19:09:23.896654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.896577    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee201c8-5c45-46ba-b56d-b996d45fcab4" containerName="revoke-keys"
Nov 03 19:09:23.896744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.896733    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee201c8-5c45-46ba-b56d-b996d45fcab4" containerName="revoke-keys"
Nov 03 19:09:23.900219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.900200    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"]
Nov 03 19:09:23.900318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:23.900274    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.032256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.032219    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\") pod \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") "
Nov 03 19:09:24.032448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.032299    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\" (UniqueName: \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\") pod \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") "
Nov 03 19:09:24.032524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.032458    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\") pod \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\" (UID: \"5ee201c8-5c45-46ba-b56d-b996d45fcab4\") "
Nov 03 19:09:24.033226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.033197    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca" pod "5ee201c8-5c45-46ba-b56d-b996d45fcab4" (UID: "5ee201c8-5c45-46ba-b56d-b996d45fcab4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:24.033394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.033254    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfnxr\" (UniqueName: \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.033591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.033568    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.034024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.034001    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.035230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.035202    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf" pod "5ee201c8-5c45-46ba-b56d-b996d45fcab4" (UID: "5ee201c8-5c45-46ba-b56d-b996d45fcab4"). InnerVolumeSpecName "kube-api-access-gjswf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:24.035343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.035310    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:24.036692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.036663    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca" pod "5ee201c8-5c45-46ba-b56d-b996d45fcab4" (UID: "5ee201c8-5c45-46ba-b56d-b996d45fcab4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:24.136908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.136876    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.137100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.137086    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfnxr\" (UniqueName: \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.137156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.137127    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.137241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.137225    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee201c8-5c45-46ba-b56d-b996d45fcab4-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:24.137304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.137247    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjswf\" (UniqueName: \"kubernetes.io/projected/5ee201c8-5c45-46ba-b56d-b996d45fcab4-kube-api-access-gjswf\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:24.137751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.137726    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.139191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.139173    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.194375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.194337    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfnxr\" (UniqueName: \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\") pod \"maas-api-revoke-sub-83776904-fljhh\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.228300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.228262    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:24.252727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.252682    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:24.608399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.608374    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"]
Nov 03 19:09:24.609959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.609902    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" podStartSLOduration=7.609889651 podStartE2EDuration="7.609889651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:24.60498593 +0000 UTC m=+1462.882339514" watchObservedRunningTime="2026-11-03 19:09:24.609889651 +0000 UTC m=+1462.887243254"
Nov 03 19:09:24.612164 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:24.612142    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod985dc89e_f210_45f9_8ab9_bcee80d6cb6b.slice/crio-8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84 WatchSource:0}: Error finding container 8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84: Status 404 returned error can't find the container with id 8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84
Nov 03 19:09:24.621351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.621324    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:24.745973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.745856    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" event={"ID":"cdf92c12-4624-4766-ac20-4b80f04dd327","Type":"ContainerStarted","Data":"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"}
Nov 03 19:09:24.746501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.746075    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" event={"ID":"cdf92c12-4624-4766-ac20-4b80f04dd327","Type":"ContainerStarted","Data":"7d91b32f394cc7a9d38e9670a0267ebf6756902f4d6660fccefb54c661fa02a6"}
Nov 03 19:09:24.746501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.746116    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:24.747862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.747831    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f" event={"ID":"5ee201c8-5c45-46ba-b56d-b996d45fcab4","Type":"ContainerDied","Data":"2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782"}
Nov 03 19:09:24.747995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.747860    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"
Nov 03 19:09:24.747995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.747883    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbb5ab623f6d04060702d5a2e0e46d322c7296566d9c18d7c803eb7321c9782"
Nov 03 19:09:24.750059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.749896    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh" event={"ID":"985dc89e-f210-45f9-8ab9-bcee80d6cb6b","Type":"ContainerStarted","Data":"8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84"}
Nov 03 19:09:24.751948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.751923    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" event={"ID":"d9c071a1-af27-4af7-b7cb-1eb434e07caf","Type":"ContainerStarted","Data":"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"}
Nov 03 19:09:24.871847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:24.871756    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:25.554206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:25.554176    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:25.761719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:25.761677    4944 generic.go:356] "Generic (PLEG): container finished" podID="985dc89e-f210-45f9-8ab9-bcee80d6cb6b" containerID="985a4042cb16fcc06d8823d51d5b9a4505d070ff03cb61465b719e4c10282dbc" exitCode=0
Nov 03 19:09:25.762167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:25.761764    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh" event={"ID":"985dc89e-f210-45f9-8ab9-bcee80d6cb6b","Type":"ContainerDied","Data":"985a4042cb16fcc06d8823d51d5b9a4505d070ff03cb61465b719e4c10282dbc"}
Nov 03 19:09:26.053234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.053194    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"]
Nov 03 19:09:26.059004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.058973    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7635a2d1-6787n"]
Nov 03 19:09:26.192783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.192753    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:26.291067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.291015    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" podStartSLOduration=9.291001008 podStartE2EDuration="9.291001008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:26.28916087 +0000 UTC m=+1464.566514453" watchObservedRunningTime="2026-11-03 19:09:26.291001008 +0000 UTC m=+1464.568354591"
Nov 03 19:09:26.439153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.438951    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:26.841251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.839510    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:09:26.844391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.844350    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:26.894522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.893928    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-e114ecf4-dockercfg-w9tt8\""
Nov 03 19:09:26.894522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.894106    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-e114ecf4\""
Nov 03 19:09:26.897224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.897193    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:09:26.898034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.898004    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:26.898179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.898160    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhrvw\" (UniqueName: \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:26.973671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.973641    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:26.999593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:26.999559    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhrvw\" (UniqueName: \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.000610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.000581    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.001542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.001519    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.058636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.058113    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhrvw\" (UniqueName: \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\") pod \"payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.102127    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\") pod \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") "
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.102172    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\" (UniqueName: \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\") pod \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") "
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.102272    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\") pod \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\" (UID: \"985dc89e-f210-45f9-8ab9-bcee80d6cb6b\") "
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.102648    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca" pod "985dc89e-f210-45f9-8ab9-bcee80d6cb6b" (UID: "985dc89e-f210-45f9-8ab9-bcee80d6cb6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.103313    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.105283    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr" pod "985dc89e-f210-45f9-8ab9-bcee80d6cb6b" (UID: "985dc89e-f210-45f9-8ab9-bcee80d6cb6b"). InnerVolumeSpecName "kube-api-access-pfnxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:27.111601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.105418    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca" pod "985dc89e-f210-45f9-8ab9-bcee80d6cb6b" (UID: "985dc89e-f210-45f9-8ab9-bcee80d6cb6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:27.163944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.163912    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.204813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.204755    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:27.204813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.204812    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfnxr\" (UniqueName: \"kubernetes.io/projected/985dc89e-f210-45f9-8ab9-bcee80d6cb6b-kube-api-access-pfnxr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:27.554384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.554346    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:09:27.557305 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:27.557278    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a77cde_1b68_46d7_b856_6054d4b27276.slice/crio-43a0f3653ff856358cb5d13c6bcf07a82dfe961e6af159ca082d1540a0e6c758 WatchSource:0}: Error finding container 43a0f3653ff856358cb5d13c6bcf07a82dfe961e6af159ca082d1540a0e6c758: Status 404 returned error can't find the container with id 43a0f3653ff856358cb5d13c6bcf07a82dfe961e6af159ca082d1540a0e6c758
Nov 03 19:09:27.781427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.781390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" event={"ID":"86a77cde-1b68-46d7-b856-6054d4b27276","Type":"ContainerStarted","Data":"701bc444b6bfcd7b67103c124b73fb52e6bc7c746829491f2149e0cb2066b084"}
Nov 03 19:09:27.781602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.781459    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:27.781602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.781483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" event={"ID":"86a77cde-1b68-46d7-b856-6054d4b27276","Type":"ContainerStarted","Data":"43a0f3653ff856358cb5d13c6bcf07a82dfe961e6af159ca082d1540a0e6c758"}
Nov 03 19:09:27.783004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.782982    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"
Nov 03 19:09:27.783115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.783023    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh" event={"ID":"985dc89e-f210-45f9-8ab9-bcee80d6cb6b","Type":"ContainerDied","Data":"8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84"}
Nov 03 19:09:27.783115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.783058    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea9cfd6d9efd5ed768c7f8aa0ecf6b4546101c7497db4c6779e3079eb749c84"
Nov 03 19:09:27.894483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.894386    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" podStartSLOduration=1.8943683789999999 podStartE2EDuration="1.894368379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:27.889177223 +0000 UTC m=+1466.166530804" watchObservedRunningTime="2026-11-03 19:09:27.894368379 +0000 UTC m=+1466.171721986"
Nov 03 19:09:27.936628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:27.936594    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80325897-670f-4d63-9ec0-1db7ad17f7cd" path="/var/lib/kubelet/pods/80325897-670f-4d63-9ec0-1db7ad17f7cd/volumes"
Nov 03 19:09:28.431028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.430986    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"]
Nov 03 19:09:28.431720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.431706    4944 cpu_manager.go:414] "containerMap: removing container" podUID="985dc89e-f210-45f9-8ab9-bcee80d6cb6b" containerName="revoke-keys"
Nov 03 19:09:28.431765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.431722    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="985dc89e-f210-45f9-8ab9-bcee80d6cb6b" containerName="revoke-keys"
Nov 03 19:09:28.431879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.431868    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="985dc89e-f210-45f9-8ab9-bcee80d6cb6b" containerName="revoke-keys"
Nov 03 19:09:28.434966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.434944    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.483859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.483782    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:09:28.483859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.483845    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:09:28.484050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.483821    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:09:28.484625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.484598    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:09:28.490445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.490420    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"]
Nov 03 19:09:28.525762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.525728    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrckj\" (UniqueName: \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.525950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.525898    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.526010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.525992    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.627216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.627181    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.627376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.627244    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.627542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.627524    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrckj\" (UniqueName: \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.627987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.627956    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.629830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.629780    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.684845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.684748    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrckj\" (UniqueName: \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\") pod \"maas-api-revoke-sub-7b780ca8-tb7qp\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:28.750000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:28.749968    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:29.131130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:29.131101    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"]
Nov 03 19:09:29.134590 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:29.134548    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce71709_d05b_4ac1_8864_a1bf0b352042.slice/crio-b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051 WatchSource:0}: Error finding container b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051: Status 404 returned error can't find the container with id b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051
Nov 03 19:09:29.574556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:29.574516    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:29.803053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:29.803021    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp" event={"ID":"1ce71709-d05b-4ac1-8864-a1bf0b352042","Type":"ContainerStarted","Data":"252c42230db3064a08ee61d3fa6bca55ff0033050ec387b8bc33bdfa681cb002"}
Nov 03 19:09:29.803234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:29.803080    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp" event={"ID":"1ce71709-d05b-4ac1-8864-a1bf0b352042","Type":"ContainerStarted","Data":"b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051"}
Nov 03 19:09:29.915144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:29.915087    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp" podStartSLOduration=1.915072874 podStartE2EDuration="1.915072874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:29.910453469 +0000 UTC m=+1468.187807051" watchObservedRunningTime="2026-11-03 19:09:29.915072874 +0000 UTC m=+1468.192426501"
Nov 03 19:09:30.812999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:30.812959    4944 generic.go:356] "Generic (PLEG): container finished" podID="1ce71709-d05b-4ac1-8864-a1bf0b352042" containerID="252c42230db3064a08ee61d3fa6bca55ff0033050ec387b8bc33bdfa681cb002" exitCode=0
Nov 03 19:09:30.813514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:30.813051    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp" event={"ID":"1ce71709-d05b-4ac1-8864-a1bf0b352042","Type":"ContainerDied","Data":"252c42230db3064a08ee61d3fa6bca55ff0033050ec387b8bc33bdfa681cb002"}
Nov 03 19:09:31.481591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.481557    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:31.612079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.612044    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"]
Nov 03 19:09:31.615771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.615752    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.674496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.674460    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"]
Nov 03 19:09:31.777209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.777168    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.777438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.777417    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jx4\" (UniqueName: \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.777483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.777464    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.879678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.879637    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.880114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.879909    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jx4\" (UniqueName: \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.880114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.879969    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.880946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.880913    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.899246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.899212    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.944774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.944738    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jx4\" (UniqueName: \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\") pod \"maas-api-revoke-sub-1b29e321-mjtd4\" (UID: \"15764425-3362-4021-966e-a939053ad959\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:31.981629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:31.981603    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:32.082960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.082872    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\") pod \"1ce71709-d05b-4ac1-8864-a1bf0b352042\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") "
Nov 03 19:09:32.083188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.083171    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\" (UniqueName: \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\") pod \"1ce71709-d05b-4ac1-8864-a1bf0b352042\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") "
Nov 03 19:09:32.083267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.083224    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\") pod \"1ce71709-d05b-4ac1-8864-a1bf0b352042\" (UID: \"1ce71709-d05b-4ac1-8864-a1bf0b352042\") "
Nov 03 19:09:32.083325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.083297    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca" pod "1ce71709-d05b-4ac1-8864-a1bf0b352042" (UID: "1ce71709-d05b-4ac1-8864-a1bf0b352042"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:32.083878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.083852    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:32.085525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.085495    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca" pod "1ce71709-d05b-4ac1-8864-a1bf0b352042" (UID: "1ce71709-d05b-4ac1-8864-a1bf0b352042"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:32.085831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.085782    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj" pod "1ce71709-d05b-4ac1-8864-a1bf0b352042" (UID: "1ce71709-d05b-4ac1-8864-a1bf0b352042"). InnerVolumeSpecName "kube-api-access-rrckj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:32.184975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.184945    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrckj\" (UniqueName: \"kubernetes.io/projected/1ce71709-d05b-4ac1-8864-a1bf0b352042-kube-api-access-rrckj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:32.184975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.184971    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ce71709-d05b-4ac1-8864-a1bf0b352042-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:32.229722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.229689    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:32.607444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.607417    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"]
Nov 03 19:09:32.611117 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:32.611086    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15764425_3362_4021_966e_a939053ad959.slice/crio-9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5 WatchSource:0}: Error finding container 9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5: Status 404 returned error can't find the container with id 9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5
Nov 03 19:09:32.714519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.714488    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:32.831176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.831144    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4" event={"ID":"15764425-3362-4021-966e-a939053ad959","Type":"ContainerStarted","Data":"9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5"}
Nov 03 19:09:32.833138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.833104    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp" event={"ID":"1ce71709-d05b-4ac1-8864-a1bf0b352042","Type":"ContainerDied","Data":"b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051"}
Nov 03 19:09:32.833262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.833160    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b407c4c573c40c6c75068518988cc4791ffba31ccb9fc06bfbc138bf45336051"
Nov 03 19:09:32.833262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.833114    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"
Nov 03 19:09:32.984383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:32.984288    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:33.232092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:33.232058    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:33.843193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:33.843162    4944 generic.go:356] "Generic (PLEG): container finished" podID="15764425-3362-4021-966e-a939053ad959" containerID="d1fd2895660a2cf6b6046b838e696f5cde7e456c392d6cb100b26d08f5a7b9cb" exitCode=0
Nov 03 19:09:33.843331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:33.843240    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4" event={"ID":"15764425-3362-4021-966e-a939053ad959","Type":"ContainerDied","Data":"d1fd2895660a2cf6b6046b838e696f5cde7e456c392d6cb100b26d08f5a7b9cb"}
Nov 03 19:09:35.011706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.011670    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:35.130427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.130333    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\") pod \"15764425-3362-4021-966e-a939053ad959\" (UID: \"15764425-3362-4021-966e-a939053ad959\") "
Nov 03 19:09:35.130620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.130493    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\" (UniqueName: \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\") pod \"15764425-3362-4021-966e-a939053ad959\" (UID: \"15764425-3362-4021-966e-a939053ad959\") "
Nov 03 19:09:35.131180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.131151    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\") pod \"15764425-3362-4021-966e-a939053ad959\" (UID: \"15764425-3362-4021-966e-a939053ad959\") "
Nov 03 19:09:35.132416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.132380    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca" pod "15764425-3362-4021-966e-a939053ad959" (UID: "15764425-3362-4021-966e-a939053ad959"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:35.135244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.135217    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca" pod "15764425-3362-4021-966e-a939053ad959" (UID: "15764425-3362-4021-966e-a939053ad959"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:35.135515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.135486    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4" pod "15764425-3362-4021-966e-a939053ad959" (UID: "15764425-3362-4021-966e-a939053ad959"). InnerVolumeSpecName "kube-api-access-q2jx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:35.225000 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:09:35.224966    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:09:35.233149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.233126    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15764425-3362-4021-966e-a939053ad959-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:35.233223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.233153    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15764425-3362-4021-966e-a939053ad959-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:35.233223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.233165    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jx4\" (UniqueName: \"kubernetes.io/projected/15764425-3362-4021-966e-a939053ad959-kube-api-access-q2jx4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:35.764610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.764578    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:09:35.764944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.764923    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:09:35.862141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.862114    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"
Nov 03 19:09:35.862304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.862146    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4" event={"ID":"15764425-3362-4021-966e-a939053ad959","Type":"ContainerDied","Data":"9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5"}
Nov 03 19:09:35.862304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:35.862192    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f053ca3550bdfa940c43012149eaae6f154f224a552989129c63310a93536e5"
Nov 03 19:09:38.000244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:38.000191    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:38.309095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:38.308997    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:38.813531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:38.804962    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:09:38.987470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:38.987422    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:39.721162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:39.721124    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:40.377813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.377568    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:40.630445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.630352    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:40.819211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.819175    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:09:40.820909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.820876    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1ce71709-d05b-4ac1-8864-a1bf0b352042" containerName="revoke-keys"
Nov 03 19:09:40.820909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.820905    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce71709-d05b-4ac1-8864-a1bf0b352042" containerName="revoke-keys"
Nov 03 19:09:40.821122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.820986    4944 cpu_manager.go:414] "containerMap: removing container" podUID="15764425-3362-4021-966e-a939053ad959" containerName="revoke-keys"
Nov 03 19:09:40.821122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.820995    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15764425-3362-4021-966e-a939053ad959" containerName="revoke-keys"
Nov 03 19:09:40.821199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.821140    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce71709-d05b-4ac1-8864-a1bf0b352042" containerName="revoke-keys"
Nov 03 19:09:40.821199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.821157    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="15764425-3362-4021-966e-a939053ad959" containerName="revoke-keys"
Nov 03 19:09:40.831040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.826313    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:40.878762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.878721    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:09:40.915966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.915778    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c2fc8901\""
Nov 03 19:09:40.915966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.915898    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c2fc8901-dockercfg-26t9z\""
Nov 03 19:09:40.922964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.922932    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq5b\" (UniqueName: \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:40.923126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:40.922998    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.000976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.000905    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:09:41.005336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.005313    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.024782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.024752    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6x5m\" (UniqueName: \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.024935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.024905    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.025079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.025059    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq5b\" (UniqueName: \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.025158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.025141    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.025758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.025741    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.052425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.052071    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:09:41.089737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.089700    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq5b\" (UniqueName: \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\") pod \"payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") " pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.129546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.129210    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.129546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.129545    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6x5m\" (UniqueName: \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.130849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.130819    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.156813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.153244    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.198922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.198762    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6x5m\" (UniqueName: \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\") pod \"payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.323831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.321047    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:41.624047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.624010    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:09:41.629043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.629014    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:41.630673 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:41.630635    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739f1cef_139e_4df6_b99f_52acabd6f48e.slice/crio-abdc1085728ae9556cb15d78c8aa6f1f9d4127fa5f0de36e44b2fbb57e4e787c WatchSource:0}: Error finding container abdc1085728ae9556cb15d78c8aa6f1f9d4127fa5f0de36e44b2fbb57e4e787c: Status 404 returned error can't find the container with id abdc1085728ae9556cb15d78c8aa6f1f9d4127fa5f0de36e44b2fbb57e4e787c
Nov 03 19:09:41.733091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.733066    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:09:41.735493 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:41.735470    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617920f7_62ac_4c2b_892e_182c5aa34bc2.slice/crio-d89b30cf237a3ef8afb58549e893494e0c660ce0eaeeb29e0ffc9a7cb64b8089 WatchSource:0}: Error finding container d89b30cf237a3ef8afb58549e893494e0c660ce0eaeeb29e0ffc9a7cb64b8089: Status 404 returned error can't find the container with id d89b30cf237a3ef8afb58549e893494e0c660ce0eaeeb29e0ffc9a7cb64b8089
Nov 03 19:09:41.895195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.895092    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:41.921046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.921010    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" event={"ID":"739f1cef-139e-4df6-b99f-52acabd6f48e","Type":"ContainerStarted","Data":"d643be3d78ddde184e1d8fcf96ca628a792fb7cfa599c2364baf55b57576aafb"}
Nov 03 19:09:41.921241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.921069    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" event={"ID":"739f1cef-139e-4df6-b99f-52acabd6f48e","Type":"ContainerStarted","Data":"abdc1085728ae9556cb15d78c8aa6f1f9d4127fa5f0de36e44b2fbb57e4e787c"}
Nov 03 19:09:41.921241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.921101    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:41.922734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.922706    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" event={"ID":"617920f7-62ac-4c2b-892e-182c5aa34bc2","Type":"ContainerStarted","Data":"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"}
Nov 03 19:09:41.922878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.922753    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" event={"ID":"617920f7-62ac-4c2b-892e-182c5aa34bc2","Type":"ContainerStarted","Data":"d89b30cf237a3ef8afb58549e893494e0c660ce0eaeeb29e0ffc9a7cb64b8089"}
Nov 03 19:09:41.922878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:41.922778    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:42.032398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:42.032339    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" podStartSLOduration=2.032321333 podStartE2EDuration="2.032321333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:42.031974356 +0000 UTC m=+1480.309327940" watchObservedRunningTime="2026-11-03 19:09:42.032321333 +0000 UTC m=+1480.309674918"
Nov 03 19:09:42.152176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:42.152061    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" podStartSLOduration=2.152041656 podStartE2EDuration="2.152041656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:42.151009534 +0000 UTC m=+1480.428363130" watchObservedRunningTime="2026-11-03 19:09:42.152041656 +0000 UTC m=+1480.429395240"
Nov 03 19:09:43.367727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:43.367691    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:43.618136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:43.618035    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:43.977651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:43.977549    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:44.227620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:44.227587    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:44.823051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:44.823012    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:45.069162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:45.069128    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:47.020917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.020878    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"]
Nov 03 19:09:47.027018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.026998    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.080278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.080239    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"]
Nov 03 19:09:47.120839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.120306    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:09:47.120839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.120449    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:09:47.120839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.120609    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:09:47.124497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.124472    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:09:47.222272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.222232    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.222472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.222297    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7pn\" (UniqueName: \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.222472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.222399    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.295307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.294999    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:47.324651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.324594    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.325842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.325490    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7pn\" (UniqueName: \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.326110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.326086    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.327095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.327067    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.329427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.329404    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.383446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.383412    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7pn\" (UniqueName: \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\") pod \"maas-api-revoke-sub-a149041e-zdrpr\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.540975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.540923    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:47.644110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.644010    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:47.752812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.739928    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:09:47.756108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.756070    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:47.811811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.805030    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:09:47.811811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.807437    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7a8bab4a\""
Nov 03 19:09:47.846154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.841098    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7a8bab4a-dockercfg-jtfwn\""
Nov 03 19:09:47.888221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.888167    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:09:47.894060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.894033    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:47.946711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.946667    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlnj\" (UniqueName: \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:47.947153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.947126    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:47.954782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:47.954749    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:09:48.049068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.049028    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlrxm\" (UniqueName: \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.049558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.049108    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.049558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.049392    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlnj\" (UniqueName: \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:48.049558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.049524    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:48.050265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.050245    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:48.107903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.107863    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlnj\" (UniqueName: \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\") pod \"payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") " pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:48.110353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.110285    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:48.145651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.145514    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"]
Nov 03 19:09:48.148341 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:48.148310    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412258e2_cc67_43f7_998f_28c95092e9cf.slice/crio-fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd WatchSource:0}: Error finding container fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd: Status 404 returned error can't find the container with id fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd
Nov 03 19:09:48.153397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.152008    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlrxm\" (UniqueName: \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.153397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.152112    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.154705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.154673    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.219894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.219694    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlrxm\" (UniqueName: \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\") pod \"payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.370302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.370264    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:48.514240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.514203    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:48.534526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.534499    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:09:48.897712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:48.897499    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:09:48.901082 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:09:48.901053    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00977ece_f9ab_4316_9316_6e13de18bd0b.slice/crio-05ea40b8dc3a88c100c77053e61163068f5629cdeebd0a41a95f9379ad67c5da WatchSource:0}: Error finding container 05ea40b8dc3a88c100c77053e61163068f5629cdeebd0a41a95f9379ad67c5da: Status 404 returned error can't find the container with id 05ea40b8dc3a88c100c77053e61163068f5629cdeebd0a41a95f9379ad67c5da
Nov 03 19:09:49.001298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.001266    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" event={"ID":"00977ece-f9ab-4316-9316-6e13de18bd0b","Type":"ContainerStarted","Data":"83434b4f8bcc323af135734549650c9d3bbccd36f12b9f90331f7caf8d73334a"}
Nov 03 19:09:49.001439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.001324    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:09:49.001439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.001350    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" event={"ID":"00977ece-f9ab-4316-9316-6e13de18bd0b","Type":"ContainerStarted","Data":"05ea40b8dc3a88c100c77053e61163068f5629cdeebd0a41a95f9379ad67c5da"}
Nov 03 19:09:49.002965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.002939    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" event={"ID":"ffb3ffda-0fa6-468c-b82d-602474b89c97","Type":"ContainerStarted","Data":"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"}
Nov 03 19:09:49.003058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.002987    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" event={"ID":"ffb3ffda-0fa6-468c-b82d-602474b89c97","Type":"ContainerStarted","Data":"574604d351c3acbc2f4449acd8ba58cc69ad1f3d30f4a9f43553f8f25bfe449f"}
Nov 03 19:09:49.003058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.003046    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:09:49.004641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.004617    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr" event={"ID":"412258e2-cc67-43f7-998f-28c95092e9cf","Type":"ContainerStarted","Data":"edd4d0ef5d64f60208fb691e5bd8520aab08800359b05955ca9e6f11890993a1"}
Nov 03 19:09:49.004730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.004662    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr" event={"ID":"412258e2-cc67-43f7-998f-28c95092e9cf","Type":"ContainerStarted","Data":"fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd"}
Nov 03 19:09:49.115478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.115423    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" podStartSLOduration=2.115409731 podStartE2EDuration="2.115409731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:49.109689734 +0000 UTC m=+1487.387043320" watchObservedRunningTime="2026-11-03 19:09:49.115409731 +0000 UTC m=+1487.392763314"
Nov 03 19:09:49.232675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.232573    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr" podStartSLOduration=3.232556453 podStartE2EDuration="3.232556453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:49.226968279 +0000 UTC m=+1487.504321863" watchObservedRunningTime="2026-11-03 19:09:49.232556453 +0000 UTC m=+1487.509910035"
Nov 03 19:09:49.341524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:49.341466    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" podStartSLOduration=2.341450742 podStartE2EDuration="2.341450742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:09:49.338157658 +0000 UTC m=+1487.615511243" watchObservedRunningTime="2026-11-03 19:09:49.341450742 +0000 UTC m=+1487.618804324"
Nov 03 19:09:50.018891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:50.018855    4944 generic.go:356] "Generic (PLEG): container finished" podID="412258e2-cc67-43f7-998f-28c95092e9cf" containerID="edd4d0ef5d64f60208fb691e5bd8520aab08800359b05955ca9e6f11890993a1" exitCode=0
Nov 03 19:09:50.019096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:50.018936    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr" event={"ID":"412258e2-cc67-43f7-998f-28c95092e9cf","Type":"ContainerDied","Data":"edd4d0ef5d64f60208fb691e5bd8520aab08800359b05955ca9e6f11890993a1"}
Nov 03 19:09:51.192025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.192000    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:51.313266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.313185    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\") pod \"412258e2-cc67-43f7-998f-28c95092e9cf\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") "
Nov 03 19:09:51.313266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.313240    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\" (UniqueName: \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\") pod \"412258e2-cc67-43f7-998f-28c95092e9cf\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") "
Nov 03 19:09:51.313483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.313434    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\") pod \"412258e2-cc67-43f7-998f-28c95092e9cf\" (UID: \"412258e2-cc67-43f7-998f-28c95092e9cf\") "
Nov 03 19:09:51.314196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.313634    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca" pod "412258e2-cc67-43f7-998f-28c95092e9cf" (UID: "412258e2-cc67-43f7-998f-28c95092e9cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:09:51.314196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.314131    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:51.316116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.316089    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca" pod "412258e2-cc67-43f7-998f-28c95092e9cf" (UID: "412258e2-cc67-43f7-998f-28c95092e9cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:09:51.316236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.316123    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn" pod "412258e2-cc67-43f7-998f-28c95092e9cf" (UID: "412258e2-cc67-43f7-998f-28c95092e9cf"). InnerVolumeSpecName "kube-api-access-zg7pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:09:51.415280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.415250    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg7pn\" (UniqueName: \"kubernetes.io/projected/412258e2-cc67-43f7-998f-28c95092e9cf-kube-api-access-zg7pn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:51.415280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:51.415281    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/412258e2-cc67-43f7-998f-28c95092e9cf-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:09:52.069216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:52.069138    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr" event={"ID":"412258e2-cc67-43f7-998f-28c95092e9cf","Type":"ContainerDied","Data":"fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd"}
Nov 03 19:09:52.069216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:52.069204    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd11228db10336217ffb0d3866a0005b51eaae18efd2f8db09574850314efddd"
Nov 03 19:09:52.069486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:52.069232    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"
Nov 03 19:09:52.143325 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:09:52.143286    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412258e2_cc67_43f7_998f_28c95092e9cf.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:09:52.945010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:52.944975    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:09:52.945477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:52.945075    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:09:53.296613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:53.296576    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:57.697082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:57.697045    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:58.106503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:58.106467    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:09:58.373244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:09:58.373149    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:00.022904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:00.022872    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:10:00.023380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:00.022941    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:10:02.128885 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.128852    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:10:02.131576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.131550    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:10:02.467531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.467439    4944 scope.go:122] "RemoveContainer" containerID="c20b2af7f2e6ca030fd92a4ca2f0dc720466f441afa699646bb7ae0fe8e5aa24"
Nov 03 19:10:02.478459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.478434    4944 scope.go:122] "RemoveContainer" containerID="4d49d8bd1018f91ec1a7fba6341c8d5ab8afd77747ab0736edc2f5e8a11401bf"
Nov 03 19:10:02.491971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.491938    4944 scope.go:122] "RemoveContainer" containerID="e6d9dde9683f9d44e1ed8c2b17cc7c55f99c9c0bf9bbcd212ad2f0706d1c04b8"
Nov 03 19:10:02.501666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.501649    4944 scope.go:122] "RemoveContainer" containerID="40d86faa80eafc841a4051c2d69063ea3740186858a52092fff5bf225f2b673d"
Nov 03 19:10:02.790717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:02.790685    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:03.060034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:03.059959    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:04.076207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:04.076172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:04.414683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:04.414607    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:05.227162 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:10:05.227072    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:10:07.767016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:07.766945    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:12.246168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:12.246135    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:12.491943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:12.491905    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:12.743890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:12.743861    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:17.087231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:17.087196    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:21.854189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:21.854153    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:22.100780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:22.100744    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:24.996411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:24.996376    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:26.590625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:26.590587    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:31.122722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.122688    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:31.426432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.426354    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:10:31.427146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.427129    4944 cpu_manager.go:414] "containerMap: removing container" podUID="412258e2-cc67-43f7-998f-28c95092e9cf" containerName="revoke-keys"
Nov 03 19:10:31.427195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.427147    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412258e2-cc67-43f7-998f-28c95092e9cf" containerName="revoke-keys"
Nov 03 19:10:31.427280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.427271    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="412258e2-cc67-43f7-998f-28c95092e9cf" containerName="revoke-keys"
Nov 03 19:10:31.431745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.431726    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.482855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.482824    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.482888    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483008    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483085    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483146    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483249    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483274    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483305    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.483389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.483367    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxkw\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.534586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.534549    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:10:31.584991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.584952    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585012    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585192    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585268    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585312    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585392    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585411    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585451    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585487    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxkw\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585571    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585658    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.585880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585864    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.586033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.585921    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.586108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.586067    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.587528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.587503    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.587899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.587880    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.642053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.642010    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.642295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.642270    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxkw\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\") pod \"maas-default-gateway-openshift-default-77f556d647-d6xxk\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:31.748923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:31.748883    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:32.246348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.246313    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:10:32.247994 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:10:32.247968    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74e3f24_3038_46ec_b27e_e72e2c1ab439.slice/crio-d61fcbe1ad1fe3e9085006088702d8f83006428857b870a7a6254ca806fc4c99 WatchSource:0}: Error finding container d61fcbe1ad1fe3e9085006088702d8f83006428857b870a7a6254ca806fc4c99: Status 404 returned error can't find the container with id d61fcbe1ad1fe3e9085006088702d8f83006428857b870a7a6254ca806fc4c99
Nov 03 19:10:32.250928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.250886    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:32.251015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.250979    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:32.251052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.251026    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:32.436553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.436508    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" event={"ID":"d74e3f24-3038-46ec-b27e-e72e2c1ab439","Type":"ContainerStarted","Data":"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"}
Nov 03 19:10:32.436733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.436584    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" event={"ID":"d74e3f24-3038-46ec-b27e-e72e2c1ab439","Type":"ContainerStarted","Data":"d61fcbe1ad1fe3e9085006088702d8f83006428857b870a7a6254ca806fc4c99"}
Nov 03 19:10:32.556053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.555939    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podStartSLOduration=1.5559169000000002 podStartE2EDuration="1.5559169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:32.549122805 +0000 UTC m=+1530.826476389" watchObservedRunningTime="2026-11-03 19:10:32.5559169 +0000 UTC m=+1530.833270485"
Nov 03 19:10:32.749571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:32.749503    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:33.751267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:33.751232    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:33.751655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:33.751314    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:34.749867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:34.749834    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:34.750047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:34.749898    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:35.253387 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:10:35.253349    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:10:35.750540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:35.750508    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:35.750704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:35.750564    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:35.994426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:35.993763    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:36.320332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:36.320295    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:36.565896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:36.565825    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:36.749589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:36.749559    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:36.749778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:36.749619    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:37.750120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:37.750080    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:37.750537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:37.750155    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:38.750052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:38.750008    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:38.750208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:38.750077    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:39.750006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.749971    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 19:10:39.750225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.750045    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 19:10:39.794733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.794527    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:10:39.811735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.811700    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.861966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.861926    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-46ccd045-openshift-default-dockercfg-87s4t\""
Nov 03 19:10:39.914447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.914411    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:10:39.990837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.990803    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.990874    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.990915    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gjmn\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.990986    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.991007    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.991029    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.991174    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.991213    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:39.991326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:39.991267    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093166    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093227    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093259    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093433    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093482    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093528    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gjmn\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093619    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093646    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093685    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093713    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.093978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.093950    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.094281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.094056    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.094281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.094155    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.094449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.094316    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.096131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.096106    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.096242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.096217    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.150808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.150748    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.150953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.150775    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gjmn\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\") pod \"e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") " pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.428119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.428026    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:40.873053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.873023    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:10:40.874562 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:10:40.874530    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad813bb_37fc_4eba_9c62_59799b12467f.slice/crio-de379762d433eddf1fea7f0c1cbdda5f8f0211cc955daae00197eff5664bab1a WatchSource:0}: Error finding container de379762d433eddf1fea7f0c1cbdda5f8f0211cc955daae00197eff5664bab1a: Status 404 returned error can't find the container with id de379762d433eddf1fea7f0c1cbdda5f8f0211cc955daae00197eff5664bab1a
Nov 03 19:10:40.877633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.877594    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:40.877706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.877681    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:40.877744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:40.877730    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:41.010752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.010714    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:41.135355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.135260    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:41.137355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.136381    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:41.137355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.137182    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:10:41.550124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.550078    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" event={"ID":"9ad813bb-37fc-4eba-9c62-59799b12467f","Type":"ContainerStarted","Data":"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"}
Nov 03 19:10:41.550321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.550145    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" event={"ID":"9ad813bb-37fc-4eba-9c62-59799b12467f","Type":"ContainerStarted","Data":"de379762d433eddf1fea7f0c1cbdda5f8f0211cc955daae00197eff5664bab1a"}
Nov 03 19:10:41.664537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.664480    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podStartSLOduration=2.664466383 podStartE2EDuration="2.664466383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:41.661893733 +0000 UTC m=+1539.939247316" watchObservedRunningTime="2026-11-03 19:10:41.664466383 +0000 UTC m=+1539.941819965"
Nov 03 19:10:41.895429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:41.894681    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:42.152230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:42.152102    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:42.430955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:42.430118    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:42.432495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:42.432468    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:42.432719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:42.432685    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:43.429818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:43.429225    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:43.429818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:43.429288    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:44.430812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:44.428470    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:44.430812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:44.428534    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:45.432189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:45.432153    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:45.432597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:45.432210    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:45.615878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:45.615070    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:46.430731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.430232    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:46.430731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.430298    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:46.544967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.544930    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:10:46.549955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.549935    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.603831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.603797    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.603844    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604030    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604166 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604067    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604166 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604156    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604203    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604296    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604335    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.604463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.604446    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.646672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.646639    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:10:46.705848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705723    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.705848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705759    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.705848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705827    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705869    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705930    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.705962    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706059    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706146    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706184    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706249    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706440    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706558    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706604    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.706756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.706715    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.733097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.733066    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.734455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.734421    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.762837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.762808    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.762945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.762871    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\") pod \"maas-default-gateway-openshift-default-77f556d647-4t7v2\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:46.865177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:46.865137    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:47.338094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.337987    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:10:47.339604 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:10:47.339577    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a503d9_371d_4265_951c_2cc6b0abcf5c.slice/crio-60eddc3134ad83023aaa2609ebb052d0fe05abc6338c6e2d5f41f19658857bc4 WatchSource:0}: Error finding container 60eddc3134ad83023aaa2609ebb052d0fe05abc6338c6e2d5f41f19658857bc4: Status 404 returned error can't find the container with id 60eddc3134ad83023aaa2609ebb052d0fe05abc6338c6e2d5f41f19658857bc4
Nov 03 19:10:47.342109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.342075    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:47.342209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.342144    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:47.342209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.342177    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:10:47.429281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.429243    4944 patch_prober.go:29] interesting pod/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Nov 03 19:10:47.429463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.429323    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Nov 03 19:10:47.611168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.611070    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" event={"ID":"f8a503d9-371d-4265-951c-2cc6b0abcf5c","Type":"ContainerStarted","Data":"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"}
Nov 03 19:10:47.611168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.611131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" event={"ID":"f8a503d9-371d-4265-951c-2cc6b0abcf5c","Type":"ContainerStarted","Data":"60eddc3134ad83023aaa2609ebb052d0fe05abc6338c6e2d5f41f19658857bc4"}
Nov 03 19:10:47.740810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.740657    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podStartSLOduration=1.7406389089999998 podStartE2EDuration="1.740638909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:10:47.72907484 +0000 UTC m=+1546.006428437" watchObservedRunningTime="2026-11-03 19:10:47.740638909 +0000 UTC m=+1546.017992492"
Nov 03 19:10:47.865861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:47.865746    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:48.437120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:48.437088    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:48.437632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:48.437610    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:48.438546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:48.438525    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:10:48.867356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:48.867321    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:48.867837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:48.867372    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:49.866473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:49.866435    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:49.866688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:49.866499    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:50.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:50.773375    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:50.866600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:50.866451    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:50.866600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:50.866517    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:51.865961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:51.865925    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:51.866328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:51.865987    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:52.866399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:52.866361    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:52.866918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:52.866415    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:53.866074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:53.866033    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:53.866260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:53.866104    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:54.866065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:54.866027    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body=
Nov 03 19:10:54.866474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:54.866095    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused"
Nov 03 19:10:55.526380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:55.526338    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:55.773048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:55.773013    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:56.134284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:56.134249    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:10:56.390599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:56.390499    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:56.391489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:56.391467    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:56.392360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:56.392338    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:10:56.644067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:10:56.643966    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:00.213782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:00.213749    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:02.549801 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:02.549769    4944 scope.go:122] "RemoveContainer" containerID="3b6abac728901cf4fe5c5351772a1974fc22310d78f1eaf95841dec5443813f2"
Nov 03 19:11:02.561079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:02.561053    4944 scope.go:122] "RemoveContainer" containerID="4542d0e0a0f652b9f9c3c87d11c142c80bbf107ebbe0ddbf18303e29922ee62e"
Nov 03 19:11:02.571369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:02.571284    4944 scope.go:122] "RemoveContainer" containerID="e13e5d9e6adc1fb1b2e28f08974c22f922de2097ebbb30631bde00d98bb71ee2"
Nov 03 19:11:02.582017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:02.581997    4944 scope.go:122] "RemoveContainer" containerID="b48419bfda2c58e6adbf146129f687660b88e3f79119e578c3c66cf63cee5dae"
Nov 03 19:11:05.226821 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:11:05.226764    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:11:05.348995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:05.348963    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:10.475242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:10.475204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:11.785262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.785188    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"]
Nov 03 19:11:11.791930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.791909    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.843547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.843508    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"]
Nov 03 19:11:11.877360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.877326    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.877360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.877362    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.877567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.877383    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkzqr\" (UniqueName: \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.882990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.882964    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:11.882990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.882967    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:11.883133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.883009    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:11.883133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.883109    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:11:11.978919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.978887    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.978919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.978925    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.979165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.978959    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkzqr\" (UniqueName: \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.979654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.979630    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:11.981412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:11.981386    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:12.036011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:12.035951    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkzqr\" (UniqueName: \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\") pod \"maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:12.112212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:12.112184    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:12.488814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:12.488759    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"]
Nov 03 19:11:12.491709 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:12.491675    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3602559f_9cec_4cf0_adbc_48fcae42e824.slice/crio-655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd WatchSource:0}: Error finding container 655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd: Status 404 returned error can't find the container with id 655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd
Nov 03 19:11:12.860175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:12.860135    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd" event={"ID":"3602559f-9cec-4cf0-adbc-48fcae42e824","Type":"ContainerStarted","Data":"655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd"}
Nov 03 19:11:13.869931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:13.869899    4944 generic.go:356] "Generic (PLEG): container finished" podID="3602559f-9cec-4cf0-adbc-48fcae42e824" containerID="f7047955ff2848e98f3997804d307f60b88fee6f331f24421d701442aa6b2d2c" exitCode=0
Nov 03 19:11:13.870326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:13.869990    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd" event={"ID":"3602559f-9cec-4cf0-adbc-48fcae42e824","Type":"ContainerDied","Data":"f7047955ff2848e98f3997804d307f60b88fee6f331f24421d701442aa6b2d2c"}
Nov 03 19:11:15.056255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.056230    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:15.127086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.127047    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\" (UniqueName: \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\") pod \"3602559f-9cec-4cf0-adbc-48fcae42e824\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") "
Nov 03 19:11:15.128648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.128572    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\") pod \"3602559f-9cec-4cf0-adbc-48fcae42e824\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") "
Nov 03 19:11:15.129356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.129200    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca" pod "3602559f-9cec-4cf0-adbc-48fcae42e824" (UID: "3602559f-9cec-4cf0-adbc-48fcae42e824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:15.129698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.129677    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\") pod \"3602559f-9cec-4cf0-adbc-48fcae42e824\" (UID: \"3602559f-9cec-4cf0-adbc-48fcae42e824\") "
Nov 03 19:11:15.131926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.131902    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:15.135601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.135478    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr" pod "3602559f-9cec-4cf0-adbc-48fcae42e824" (UID: "3602559f-9cec-4cf0-adbc-48fcae42e824"). InnerVolumeSpecName "kube-api-access-nkzqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:15.135761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.135572    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca" pod "3602559f-9cec-4cf0-adbc-48fcae42e824" (UID: "3602559f-9cec-4cf0-adbc-48fcae42e824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:15.179711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.179673    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:15.234941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.234905    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkzqr\" (UniqueName: \"kubernetes.io/projected/3602559f-9cec-4cf0-adbc-48fcae42e824-kube-api-access-nkzqr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:15.234941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.234935    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3602559f-9cec-4cf0-adbc-48fcae42e824-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:15.891433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.891391    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"
Nov 03 19:11:15.891616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.891409    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd" event={"ID":"3602559f-9cec-4cf0-adbc-48fcae42e824","Type":"ContainerDied","Data":"655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd"}
Nov 03 19:11:15.891616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:15.891528    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655b0831866347240db5b4ecaf09a5a7e5a4f60ce555146a0108ba9552f0b3cd"
Nov 03 19:11:16.149839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:16.149737    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"]
Nov 03 19:11:16.156672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:16.156642    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fa09cd3-pmf6h"]
Nov 03 19:11:17.935611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:17.935576    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067b0d5c-9484-4cab-bbc8-e1fd3d27981a" path="/var/lib/kubelet/pods/067b0d5c-9484-4cab-bbc8-e1fd3d27981a/volumes"
Nov 03 19:11:19.148036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.148001    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"]
Nov 03 19:11:19.148808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.148775    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3602559f-9cec-4cf0-adbc-48fcae42e824" containerName="revoke-keys"
Nov 03 19:11:19.148873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.148810    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3602559f-9cec-4cf0-adbc-48fcae42e824" containerName="revoke-keys"
Nov 03 19:11:19.149072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.149048    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3602559f-9cec-4cf0-adbc-48fcae42e824" containerName="revoke-keys"
Nov 03 19:11:19.155133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.155112    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.206211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.206183    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"]
Nov 03 19:11:19.211206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.211171    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.211347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.211318    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmfrh\" (UniqueName: \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.211464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.211447    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.233153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.233117    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:19.245671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.245649    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:19.245767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.245740    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:19.245767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.245752    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:11:19.313085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.313052    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmfrh\" (UniqueName: \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.313264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.313159    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.313437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.313417    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.314062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.314042    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.315579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.315556    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.370640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.370601    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmfrh\" (UniqueName: \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\") pod \"maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.469271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.469191    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:19.847649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.847625    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"]
Nov 03 19:11:19.850099 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:19.850060    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5250c46_ab57_415e_b9cd_f79ca7a4c7e0.slice/crio-c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde WatchSource:0}: Error finding container c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde: Status 404 returned error can't find the container with id c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde
Nov 03 19:11:19.935777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:19.935741    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p" event={"ID":"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0","Type":"ContainerStarted","Data":"c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde"}
Nov 03 19:11:20.532820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:20.532766    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:20.937197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:20.937161    4944 generic.go:356] "Generic (PLEG): container finished" podID="d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" containerID="70e4512b7b49eff4417034619721b5b7a53a4a33793b2908ad09799b3254c331" exitCode=0
Nov 03 19:11:20.937394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:20.937251    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p" event={"ID":"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0","Type":"ContainerDied","Data":"70e4512b7b49eff4417034619721b5b7a53a4a33793b2908ad09799b3254c331"}
Nov 03 19:11:22.130987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.130961    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:22.255978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.255945    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\") pod \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") "
Nov 03 19:11:22.256166 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.256117    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\") pod \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") "
Nov 03 19:11:22.256285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.256267    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\" (UniqueName: \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\") pod \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\" (UID: \"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0\") "
Nov 03 19:11:22.256376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.256321    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca" pod "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" (UID: "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:22.257094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.257064    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:22.258295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.258270    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca" pod "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" (UID: "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:22.258904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.258882    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh" pod "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" (UID: "d5250c46-ab57-415e-b9cd-f79ca7a4c7e0"). InnerVolumeSpecName "kube-api-access-nmfrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:22.358334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.358302    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:22.358334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.358330    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmfrh\" (UniqueName: \"kubernetes.io/projected/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0-kube-api-access-nmfrh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:22.954213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.954176    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p" event={"ID":"d5250c46-ab57-415e-b9cd-f79ca7a4c7e0","Type":"ContainerDied","Data":"c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde"}
Nov 03 19:11:22.954416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.954228    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45cee545929347e421f8a70e4b3f23faf4889eafa798afdeb2f8182bc869fde"
Nov 03 19:11:22.954416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:22.954185    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"
Nov 03 19:11:23.040245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:23.040204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:23.414280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:23.414246    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:11:23.512768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:23.512725    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:11:23.866542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:23.866507    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:24.112054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:24.112021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:25.528972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:25.528937    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:25.775291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:25.775260    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:26.421437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:26.421389    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:28.355270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.355202    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:28.396940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.396906    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"]
Nov 03 19:11:28.397665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.397649    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" containerName="revoke-keys"
Nov 03 19:11:28.397715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.397666    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" containerName="revoke-keys"
Nov 03 19:11:28.397865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.397854    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" containerName="revoke-keys"
Nov 03 19:11:28.402054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.402032    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.415076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.415027    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" podUID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" containerName="payload-processing" containerID="cri-o://254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60" gracePeriod=25
Nov 03 19:11:28.454312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.454270    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:28.457758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.457729    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"]
Nov 03 19:11:28.463612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.463581    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4wl7\" (UniqueName: \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.463997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.463968    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.464103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.464019    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.494955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.494934    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:28.495316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.495289    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:11:28.495472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.495379    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:28.513522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.513493    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" podUID="cdf92c12-4624-4766-ac20-4b80f04dd327" containerName="payload-pre-processing" containerID="cri-o://48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc" gracePeriod=25
Nov 03 19:11:28.565742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.565709    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4wl7\" (UniqueName: \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.566049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.566024    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.566190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.566066    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.566718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.566691    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.569075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.569052    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.611592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.611325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:28.624973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.623921    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4wl7\" (UniqueName: \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\") pod \"maas-api-revoke-sub-1452626f-gf9bw\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.716648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.716618    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:28.722299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.722272    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:11:28.771738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.771699    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\") pod \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") "
Nov 03 19:11:28.772246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.772221    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\" (UniqueName: \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\") pod \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\" (UID: \"d9c071a1-af27-4af7-b7cb-1eb434e07caf\") "
Nov 03 19:11:28.774209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.773292    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume" pod "d9c071a1-af27-4af7-b7cb-1eb434e07caf" (UID: "d9c071a1-af27-4af7-b7cb-1eb434e07caf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:28.774740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.774711    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c071a1-af27-4af7-b7cb-1eb434e07caf-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:28.777968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.777937    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n" pod "d9c071a1-af27-4af7-b7cb-1eb434e07caf" (UID: "d9c071a1-af27-4af7-b7cb-1eb434e07caf"). InnerVolumeSpecName "kube-api-access-p956n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:28.838103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.837425    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:11:28.861778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.860743    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:28.876234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.876194    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\" (UniqueName: \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\") pod \"cdf92c12-4624-4766-ac20-4b80f04dd327\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") "
Nov 03 19:11:28.876386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.876337    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\") pod \"cdf92c12-4624-4766-ac20-4b80f04dd327\" (UID: \"cdf92c12-4624-4766-ac20-4b80f04dd327\") "
Nov 03 19:11:28.877341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.877313    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume" pod "cdf92c12-4624-4766-ac20-4b80f04dd327" (UID: "cdf92c12-4624-4766-ac20-4b80f04dd327"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:28.877652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.877634    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdf92c12-4624-4766-ac20-4b80f04dd327-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:28.877726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.877655    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p956n\" (UniqueName: \"kubernetes.io/projected/d9c071a1-af27-4af7-b7cb-1eb434e07caf-kube-api-access-p956n\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:28.879495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.879469    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn" pod "cdf92c12-4624-4766-ac20-4b80f04dd327" (UID: "cdf92c12-4624-4766-ac20-4b80f04dd327"). InnerVolumeSpecName "kube-api-access-mtccn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:28.980354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:28.980324    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtccn\" (UniqueName: \"kubernetes.io/projected/cdf92c12-4624-4766-ac20-4b80f04dd327-kube-api-access-mtccn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:29.012135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.012095    4944 generic.go:356] "Generic (PLEG): container finished" podID="cdf92c12-4624-4766-ac20-4b80f04dd327" containerID="48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc" exitCode=0
Nov 03 19:11:29.012320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.012158    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"
Nov 03 19:11:29.012320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.012188    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" event={"ID":"cdf92c12-4624-4766-ac20-4b80f04dd327","Type":"ContainerDied","Data":"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"}
Nov 03 19:11:29.012320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.012285    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw" event={"ID":"cdf92c12-4624-4766-ac20-4b80f04dd327","Type":"ContainerDied","Data":"7d91b32f394cc7a9d38e9670a0267ebf6756902f4d6660fccefb54c661fa02a6"}
Nov 03 19:11:29.012507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.012349    4944 scope.go:122] "RemoveContainer" containerID="48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"
Nov 03 19:11:29.014434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.014406    4944 generic.go:356] "Generic (PLEG): container finished" podID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" containerID="254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60" exitCode=0
Nov 03 19:11:29.014555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.014465    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" event={"ID":"d9c071a1-af27-4af7-b7cb-1eb434e07caf","Type":"ContainerDied","Data":"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"}
Nov 03 19:11:29.014555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.014505    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc" event={"ID":"d9c071a1-af27-4af7-b7cb-1eb434e07caf","Type":"ContainerDied","Data":"2a225f44d30413798f7292cecc14cad34fdb4c263de773ac68e9655ce81ddefc"}
Nov 03 19:11:29.014681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.014577    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"
Nov 03 19:11:29.026452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.026427    4944 scope.go:122] "RemoveContainer" containerID="48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"
Nov 03 19:11:29.026772 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:11:29.026749    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc\": container with ID starting with 48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc not found: ID does not exist" containerID="48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"
Nov 03 19:11:29.026872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.026780    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc"} err="failed to get container status \"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc\": rpc error: code = NotFound desc = could not find container \"48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc\": container with ID starting with 48c8a6a2c029f6b669c38000823821839beabca913de21f957419a83a9b3c0bc not found: ID does not exist"
Nov 03 19:11:29.026872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.026819    4944 scope.go:122] "RemoveContainer" containerID="254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"
Nov 03 19:11:29.036648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.036629    4944 scope.go:122] "RemoveContainer" containerID="254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"
Nov 03 19:11:29.036918 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:11:29.036899    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60\": container with ID starting with 254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60 not found: ID does not exist" containerID="254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"
Nov 03 19:11:29.036987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.036925    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60"} err="failed to get container status \"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60\": rpc error: code = NotFound desc = could not find container \"254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60\": container with ID starting with 254e16e949b80934966b60616fb035ee0841b292ffa82139fb35630d5a522d60 not found: ID does not exist"
Nov 03 19:11:29.129575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.129506    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"]
Nov 03 19:11:29.135515 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:29.135485    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244d362e_a6d0_4efb_ac35_7a4d2f80e968.slice/crio-e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6 WatchSource:0}: Error finding container e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6: Status 404 returned error can't find the container with id e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6
Nov 03 19:11:29.174398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.174375    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:11:29.183266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.183232    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-46ccd045-84cdb98b48-zl2dc"]
Nov 03 19:11:29.279088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.279054    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:11:29.284126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.284101    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-46ccd045-fcc5f8cff-kglrw"]
Nov 03 19:11:29.936671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.936632    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf92c12-4624-4766-ac20-4b80f04dd327" path="/var/lib/kubelet/pods/cdf92c12-4624-4766-ac20-4b80f04dd327/volumes"
Nov 03 19:11:29.937332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:29.937301    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" path="/var/lib/kubelet/pods/d9c071a1-af27-4af7-b7cb-1eb434e07caf/volumes"
Nov 03 19:11:30.026365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.026333    4944 generic.go:356] "Generic (PLEG): container finished" podID="244d362e-a6d0-4efb-ac35-7a4d2f80e968" containerID="62895844b5fe7e37adf705c4a862a0a7e11fa24f6917130682f395d011ed8ce9" exitCode=0
Nov 03 19:11:30.026529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.026418    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw" event={"ID":"244d362e-a6d0-4efb-ac35-7a4d2f80e968","Type":"ContainerDied","Data":"62895844b5fe7e37adf705c4a862a0a7e11fa24f6917130682f395d011ed8ce9"}
Nov 03 19:11:30.026529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.026477    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw" event={"ID":"244d362e-a6d0-4efb-ac35-7a4d2f80e968","Type":"ContainerStarted","Data":"e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6"}
Nov 03 19:11:30.050535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.050503    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"]
Nov 03 19:11:30.074234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.074206    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:30.096577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.096545    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-424ed2af-kzmtj"]
Nov 03 19:11:30.917606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:30.917569    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:11:31.276840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.276764    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:31.310589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.310559    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:31.421390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.421362    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\") pod \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") "
Nov 03 19:11:31.421548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.421456    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\" (UniqueName: \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\") pod \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") "
Nov 03 19:11:31.421548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.421542    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\") pod \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\" (UID: \"244d362e-a6d0-4efb-ac35-7a4d2f80e968\") "
Nov 03 19:11:31.423956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.422049    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca" pod "244d362e-a6d0-4efb-ac35-7a4d2f80e968" (UID: "244d362e-a6d0-4efb-ac35-7a4d2f80e968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:31.423956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.422278    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:31.424869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.424841    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7" pod "244d362e-a6d0-4efb-ac35-7a4d2f80e968" (UID: "244d362e-a6d0-4efb-ac35-7a4d2f80e968"). InnerVolumeSpecName "kube-api-access-t4wl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:31.425839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.425077    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca" pod "244d362e-a6d0-4efb-ac35-7a4d2f80e968" (UID: "244d362e-a6d0-4efb-ac35-7a4d2f80e968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:31.487919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.487879    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"]
Nov 03 19:11:31.489148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489119    4944 cpu_manager.go:414] "containerMap: removing container" podUID="244d362e-a6d0-4efb-ac35-7a4d2f80e968" containerName="revoke-keys"
Nov 03 19:11:31.489148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489145    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244d362e-a6d0-4efb-ac35-7a4d2f80e968" containerName="revoke-keys"
Nov 03 19:11:31.489354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489202    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" containerName="payload-processing"
Nov 03 19:11:31.489354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489211    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" containerName="payload-processing"
Nov 03 19:11:31.489354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489223    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cdf92c12-4624-4766-ac20-4b80f04dd327" containerName="payload-pre-processing"
Nov 03 19:11:31.489354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489231    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf92c12-4624-4766-ac20-4b80f04dd327" containerName="payload-pre-processing"
Nov 03 19:11:31.489562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489407    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c071a1-af27-4af7-b7cb-1eb434e07caf" containerName="payload-processing"
Nov 03 19:11:31.489562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489424    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf92c12-4624-4766-ac20-4b80f04dd327" containerName="payload-pre-processing"
Nov 03 19:11:31.489562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.489446    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="244d362e-a6d0-4efb-ac35-7a4d2f80e968" containerName="revoke-keys"
Nov 03 19:11:31.496591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.495349    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.524038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.523955    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdn59\" (UniqueName: \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.524196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.524124    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.524196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.524170    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.524342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.524321    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244d362e-a6d0-4efb-ac35-7a4d2f80e968-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:31.524505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.524485    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4wl7\" (UniqueName: \"kubernetes.io/projected/244d362e-a6d0-4efb-ac35-7a4d2f80e968-kube-api-access-t4wl7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:31.525373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.525324    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:31.542746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.542714    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"]
Nov 03 19:11:31.625956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.625920    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.626121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.625971    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.626165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.626152    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdn59\" (UniqueName: \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.626699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.626677    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.628635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.628610    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.687508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.687480    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdn59\" (UniqueName: \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\") pod \"maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.817516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.817428    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:31.937445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:31.937407    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4960f29-fa8c-4514-80ff-9f927d04ea94" path="/var/lib/kubelet/pods/f4960f29-fa8c-4514-80ff-9f927d04ea94/volumes"
Nov 03 19:11:32.058526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:32.058498    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"
Nov 03 19:11:32.058697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:32.058496    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw" event={"ID":"244d362e-a6d0-4efb-ac35-7a4d2f80e968","Type":"ContainerDied","Data":"e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6"}
Nov 03 19:11:32.058697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:32.058631    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ba4ce31c1f86344813a5d712240a149f5734d8f5f1ec031c49ff42795cb6e6"
Nov 03 19:11:32.194426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:32.194401    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"]
Nov 03 19:11:32.197179 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:32.197151    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c80e97_a106_446a_9a38_c1ff68bf1c0a.slice/crio-2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304 WatchSource:0}: Error finding container 2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304: Status 404 returned error can't find the container with id 2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304
Nov 03 19:11:33.071681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:33.071640    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q" event={"ID":"b6c80e97-a106-446a-9a38-c1ff68bf1c0a","Type":"ContainerStarted","Data":"d9807aa27bb8a4ac8f5f21752d13d5c3e63be2f34d0468904095656aba872aa8"}
Nov 03 19:11:33.072177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:33.071707    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q" event={"ID":"b6c80e97-a106-446a-9a38-c1ff68bf1c0a","Type":"ContainerStarted","Data":"2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304"}
Nov 03 19:11:33.186448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:33.186399    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q" podStartSLOduration=2.186385682 podStartE2EDuration="2.186385682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:33.180733214 +0000 UTC m=+1591.458086798" watchObservedRunningTime="2026-11-03 19:11:33.186385682 +0000 UTC m=+1591.463739264"
Nov 03 19:11:34.082004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:34.081972    4944 generic.go:356] "Generic (PLEG): container finished" podID="b6c80e97-a106-446a-9a38-c1ff68bf1c0a" containerID="d9807aa27bb8a4ac8f5f21752d13d5c3e63be2f34d0468904095656aba872aa8" exitCode=0
Nov 03 19:11:34.082387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:34.082052    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q" event={"ID":"b6c80e97-a106-446a-9a38-c1ff68bf1c0a","Type":"ContainerDied","Data":"d9807aa27bb8a4ac8f5f21752d13d5c3e63be2f34d0468904095656aba872aa8"}
Nov 03 19:11:34.679252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:34.679218    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:34.919838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:34.919809    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:35.172349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.172314    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:35.232937 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:11:35.232892    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:11:35.268718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.268691    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:35.397366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.397280    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\" (UniqueName: \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\") pod \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") "
Nov 03 19:11:35.397526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.397373    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\") pod \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") "
Nov 03 19:11:35.397712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.397689    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\") pod \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\" (UID: \"b6c80e97-a106-446a-9a38-c1ff68bf1c0a\") "
Nov 03 19:11:35.397860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.397706    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca" pod "b6c80e97-a106-446a-9a38-c1ff68bf1c0a" (UID: "b6c80e97-a106-446a-9a38-c1ff68bf1c0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:35.398445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.398421    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:35.399831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.399765    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca" pod "b6c80e97-a106-446a-9a38-c1ff68bf1c0a" (UID: "b6c80e97-a106-446a-9a38-c1ff68bf1c0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:35.401082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.401056    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59" pod "b6c80e97-a106-446a-9a38-c1ff68bf1c0a" (UID: "b6c80e97-a106-446a-9a38-c1ff68bf1c0a"). InnerVolumeSpecName "kube-api-access-sdn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:35.500462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.500420    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:35.500462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.500459    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdn59\" (UniqueName: \"kubernetes.io/projected/b6c80e97-a106-446a-9a38-c1ff68bf1c0a-kube-api-access-sdn59\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:35.669867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.668337    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:11:35.679814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.679031    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b6c80e97-a106-446a-9a38-c1ff68bf1c0a" containerName="revoke-keys"
Nov 03 19:11:35.679814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.679062    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c80e97-a106-446a-9a38-c1ff68bf1c0a" containerName="revoke-keys"
Nov 03 19:11:35.679814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.679339    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c80e97-a106-446a-9a38-c1ff68bf1c0a" containerName="revoke-keys"
Nov 03 19:11:35.688907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.685629    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.734190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.734156    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-144890-openshift-default-dockercfg-cxtr6\""
Nov 03 19:11:35.789158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.789122    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:11:35.820655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.820608    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxv54\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.820959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.820776    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.820977    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821009    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821041    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821079    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821307    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.821400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.821385    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.910534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.910496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:35.918360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.918244    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" podUID="86a77cde-1b68-46d7-b856-6054d4b27276" containerName="payload-pre-processing" containerID="cri-o://701bc444b6bfcd7b67103c124b73fb52e6bc7c746829491f2149e0cb2066b084" gracePeriod=25
Nov 03 19:11:35.923886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.923797    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.923886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.923850    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.923886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.923873    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.923902    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.923958    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924069    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924122    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924173    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxv54\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.924267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924235    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.925139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924645    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.925139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.924669    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.925139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.925110    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.925883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.925857    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.928359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.928331    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.930040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.930010    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.930134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.930048    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.981929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.981901    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:35.982077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:35.982057    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxv54\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\") pod \"e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") " pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:36.008709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.008668    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:36.137718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.137675    4944 generic.go:356] "Generic (PLEG): container finished" podID="86a77cde-1b68-46d7-b856-6054d4b27276" containerID="701bc444b6bfcd7b67103c124b73fb52e6bc7c746829491f2149e0cb2066b084" exitCode=0
Nov 03 19:11:36.138051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.138030    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" event={"ID":"86a77cde-1b68-46d7-b856-6054d4b27276","Type":"ContainerDied","Data":"701bc444b6bfcd7b67103c124b73fb52e6bc7c746829491f2149e0cb2066b084"}
Nov 03 19:11:36.142388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.142257    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q" event={"ID":"b6c80e97-a106-446a-9a38-c1ff68bf1c0a","Type":"ContainerDied","Data":"2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304"}
Nov 03 19:11:36.142388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.142311    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cd48228e379efb8a4dc5c6dd7da65ccec691a91724d951641841b7edef21304"
Nov 03 19:11:36.143468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.142627    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"
Nov 03 19:11:36.334840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.334813    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:11:36.432742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.432711    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\") pod \"86a77cde-1b68-46d7-b856-6054d4b27276\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") "
Nov 03 19:11:36.432966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.432855    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\" (UniqueName: \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\") pod \"86a77cde-1b68-46d7-b856-6054d4b27276\" (UID: \"86a77cde-1b68-46d7-b856-6054d4b27276\") "
Nov 03 19:11:36.433176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.433142    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume" pod "86a77cde-1b68-46d7-b856-6054d4b27276" (UID: "86a77cde-1b68-46d7-b856-6054d4b27276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:36.433551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.433532    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a77cde-1b68-46d7-b856-6054d4b27276-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:36.435079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.435052    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw" pod "86a77cde-1b68-46d7-b856-6054d4b27276" (UID: "86a77cde-1b68-46d7-b856-6054d4b27276"). InnerVolumeSpecName "kube-api-access-vhrvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:36.460049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.460015    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:11:36.464924 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:36.464877    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5bd611_ebd4_41fa_8ddc_cc519a63b0b1.slice/crio-95c54bec17052301bfc10fa18a048df2a05b251cb207467c6926bb5f70809f4f WatchSource:0}: Error finding container 95c54bec17052301bfc10fa18a048df2a05b251cb207467c6926bb5f70809f4f: Status 404 returned error can't find the container with id 95c54bec17052301bfc10fa18a048df2a05b251cb207467c6926bb5f70809f4f
Nov 03 19:11:36.468480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.468442    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:11:36.468563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.468534    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:11:36.468600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.468582    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:11:36.535467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.535441    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhrvw\" (UniqueName: \"kubernetes.io/projected/86a77cde-1b68-46d7-b856-6054d4b27276-kube-api-access-vhrvw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:36.816893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:36.816806    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:37.154155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.154068    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"
Nov 03 19:11:37.154310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.154069    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688" event={"ID":"86a77cde-1b68-46d7-b856-6054d4b27276","Type":"ContainerDied","Data":"43a0f3653ff856358cb5d13c6bcf07a82dfe961e6af159ca082d1540a0e6c758"}
Nov 03 19:11:37.154310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.154235    4944 scope.go:122] "RemoveContainer" containerID="701bc444b6bfcd7b67103c124b73fb52e6bc7c746829491f2149e0cb2066b084"
Nov 03 19:11:37.156201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.156174    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerStarted","Data":"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"}
Nov 03 19:11:37.156329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.156222    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerStarted","Data":"95c54bec17052301bfc10fa18a048df2a05b251cb207467c6926bb5f70809f4f"}
Nov 03 19:11:37.270273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.270223    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:37.319984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.319950    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:11:37.325587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.325557    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-e114ecf4-76db5656cf-b6688"]
Nov 03 19:11:37.432498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.432367    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podStartSLOduration=2.432346641 podStartE2EDuration="2.432346641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:37.425771838 +0000 UTC m=+1595.703125422" watchObservedRunningTime="2026-11-03 19:11:37.432346641 +0000 UTC m=+1595.709700226"
Nov 03 19:11:37.937205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:37.937171    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86a77cde-1b68-46d7-b856-6054d4b27276" path="/var/lib/kubelet/pods/86a77cde-1b68-46d7-b856-6054d4b27276/volumes"
Nov 03 19:11:38.009515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:38.009478    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:38.015350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:38.015321    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:38.176317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:38.176273    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:38.177706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:38.177679    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:11:39.123165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:39.123132    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:40.359883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:40.359825    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:40.611202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:40.611119    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:42.504409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:42.504374    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:44.048190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.048152    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"]
Nov 03 19:11:44.094328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.094285    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbbe146a-bhxzv"]
Nov 03 19:11:44.105879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.105840    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:11:44.209504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.209464    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:11:44.566078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.566040    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:44.813208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:44.813128    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:45.183633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:45.183597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:45.949556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:45.949513    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac41f8a9-d959-4fc4-8328-2e093eae4bf6" path="/var/lib/kubelet/pods/ac41f8a9-d959-4fc4-8328-2e093eae4bf6/volumes"
Nov 03 19:11:46.659730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:46.659692    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:47.457065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:47.457029    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:47.740112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:47.740080    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:48.182599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:48.182511    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:49.106938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.106883    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" podUID="739f1cef-139e-4df6-b99f-52acabd6f48e" containerName="payload-processing" containerID="cri-o://d643be3d78ddde184e1d8fcf96ca628a792fb7cfa599c2364baf55b57576aafb" gracePeriod=25
Nov 03 19:11:49.209735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.209693    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" podUID="617920f7-62ac-4c2b-892e-182c5aa34bc2" containerName="payload-pre-processing" containerID="cri-o://3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b" gracePeriod=25
Nov 03 19:11:49.277233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.277198    4944 generic.go:356] "Generic (PLEG): container finished" podID="739f1cef-139e-4df6-b99f-52acabd6f48e" containerID="d643be3d78ddde184e1d8fcf96ca628a792fb7cfa599c2364baf55b57576aafb" exitCode=0
Nov 03 19:11:49.277365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.277268    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" event={"ID":"739f1cef-139e-4df6-b99f-52acabd6f48e","Type":"ContainerDied","Data":"d643be3d78ddde184e1d8fcf96ca628a792fb7cfa599c2364baf55b57576aafb"}
Nov 03 19:11:49.401935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.400526    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:11:49.412072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.412039    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\" (UniqueName: \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\") pod \"739f1cef-139e-4df6-b99f-52acabd6f48e\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") "
Nov 03 19:11:49.412333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.412317    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\") pod \"739f1cef-139e-4df6-b99f-52acabd6f48e\" (UID: \"739f1cef-139e-4df6-b99f-52acabd6f48e\") "
Nov 03 19:11:49.412782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.412750    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume" pod "739f1cef-139e-4df6-b99f-52acabd6f48e" (UID: "739f1cef-139e-4df6-b99f-52acabd6f48e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:49.413122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.413097    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/739f1cef-139e-4df6-b99f-52acabd6f48e-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:49.415317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.415289    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b" pod "739f1cef-139e-4df6-b99f-52acabd6f48e" (UID: "739f1cef-139e-4df6-b99f-52acabd6f48e"). InnerVolumeSpecName "kube-api-access-qqq5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:49.498480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.498455    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:11:49.514089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.514059    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\" (UniqueName: \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\") pod \"617920f7-62ac-4c2b-892e-182c5aa34bc2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") "
Nov 03 19:11:49.514333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.514313    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\") pod \"617920f7-62ac-4c2b-892e-182c5aa34bc2\" (UID: \"617920f7-62ac-4c2b-892e-182c5aa34bc2\") "
Nov 03 19:11:49.514965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.514941    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq5b\" (UniqueName: \"kubernetes.io/projected/739f1cef-139e-4df6-b99f-52acabd6f48e-kube-api-access-qqq5b\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:49.515298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.515275    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume" pod "617920f7-62ac-4c2b-892e-182c5aa34bc2" (UID: "617920f7-62ac-4c2b-892e-182c5aa34bc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:49.517843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.517778    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m" pod "617920f7-62ac-4c2b-892e-182c5aa34bc2" (UID: "617920f7-62ac-4c2b-892e-182c5aa34bc2"). InnerVolumeSpecName "kube-api-access-q6x5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:49.616519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.616437    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/617920f7-62ac-4c2b-892e-182c5aa34bc2-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:49.616519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:49.616470    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6x5m\" (UniqueName: \"kubernetes.io/projected/617920f7-62ac-4c2b-892e-182c5aa34bc2-kube-api-access-q6x5m\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:50.288270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.288238    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"
Nov 03 19:11:50.288823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.288273    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv" event={"ID":"739f1cef-139e-4df6-b99f-52acabd6f48e","Type":"ContainerDied","Data":"abdc1085728ae9556cb15d78c8aa6f1f9d4127fa5f0de36e44b2fbb57e4e787c"}
Nov 03 19:11:50.288823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.288348    4944 scope.go:122] "RemoveContainer" containerID="d643be3d78ddde184e1d8fcf96ca628a792fb7cfa599c2364baf55b57576aafb"
Nov 03 19:11:50.290047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.290018    4944 generic.go:356] "Generic (PLEG): container finished" podID="617920f7-62ac-4c2b-892e-182c5aa34bc2" containerID="3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b" exitCode=0
Nov 03 19:11:50.290161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.290064    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"
Nov 03 19:11:50.290161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.290098    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" event={"ID":"617920f7-62ac-4c2b-892e-182c5aa34bc2","Type":"ContainerDied","Data":"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"}
Nov 03 19:11:50.290161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.290141    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2" event={"ID":"617920f7-62ac-4c2b-892e-182c5aa34bc2","Type":"ContainerDied","Data":"d89b30cf237a3ef8afb58549e893494e0c660ce0eaeeb29e0ffc9a7cb64b8089"}
Nov 03 19:11:50.299900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.299883    4944 scope.go:122] "RemoveContainer" containerID="3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"
Nov 03 19:11:50.310732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.310036    4944 scope.go:122] "RemoveContainer" containerID="3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"
Nov 03 19:11:50.314938 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:11:50.314900    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b\": container with ID starting with 3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b not found: ID does not exist" containerID="3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"
Nov 03 19:11:50.315049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.314931    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b"} err="failed to get container status \"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b\": rpc error: code = NotFound desc = could not find container \"3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b\": container with ID starting with 3a9c065e143479dc61b22ad95fdd47ff5ec584642a40a425b1e8e93dc5a7e77b not found: ID does not exist"
Nov 03 19:11:50.444910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.444882    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:11:50.450097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.450072    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c2fc8901-566b784fdb-fl5mv"]
Nov 03 19:11:50.593425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.593345    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:11:50.598592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.598560    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c2fc8901-c6bf6685f-hqmg2"]
Nov 03 19:11:50.851862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:50.847211    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:51.059522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.058139    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"]
Nov 03 19:11:51.085268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.085230    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4a55ccc-h86vm"]
Nov 03 19:11:51.096798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.096735    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:51.145659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.145570    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:11:51.146535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146508    4944 cpu_manager.go:414] "containerMap: removing container" podUID="86a77cde-1b68-46d7-b856-6054d4b27276" containerName="payload-pre-processing"
Nov 03 19:11:51.146535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146533    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86a77cde-1b68-46d7-b856-6054d4b27276" containerName="payload-pre-processing"
Nov 03 19:11:51.146724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146545    4944 cpu_manager.go:414] "containerMap: removing container" podUID="617920f7-62ac-4c2b-892e-182c5aa34bc2" containerName="payload-pre-processing"
Nov 03 19:11:51.146724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146553    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617920f7-62ac-4c2b-892e-182c5aa34bc2" containerName="payload-pre-processing"
Nov 03 19:11:51.146724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146579    4944 cpu_manager.go:414] "containerMap: removing container" podUID="739f1cef-139e-4df6-b99f-52acabd6f48e" containerName="payload-processing"
Nov 03 19:11:51.146724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146586    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739f1cef-139e-4df6-b99f-52acabd6f48e" containerName="payload-processing"
Nov 03 19:11:51.146946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146823    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="86a77cde-1b68-46d7-b856-6054d4b27276" containerName="payload-pre-processing"
Nov 03 19:11:51.146946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146839    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="617920f7-62ac-4c2b-892e-182c5aa34bc2" containerName="payload-pre-processing"
Nov 03 19:11:51.146946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.146851    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="739f1cef-139e-4df6-b99f-52acabd6f48e" containerName="payload-processing"
Nov 03 19:11:51.152053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.152032    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.201516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.201206    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-144890\""
Nov 03 19:11:51.201516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.201297    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-144890-dockercfg-rzvl2\""
Nov 03 19:11:51.210833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.210801    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:11:51.267026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.266889    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdkxs\" (UniqueName: \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.267516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.267486    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.315514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.315480    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:11:51.320457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.320432    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.373359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.370674    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.373359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.370833    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7gl\" (UniqueName: \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.373359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.371041    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.373359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.371115    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdkxs\" (UniqueName: \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.373359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.372490    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.376857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.376782    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:11:51.441603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.441516    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdkxs\" (UniqueName: \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\") pod \"payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.475856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.474614    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7gl\" (UniqueName: \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.476319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.476289    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.484968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.475286    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:51.485140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.477361    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.533213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.533177    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7gl\" (UniqueName: \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\") pod \"payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.636201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.636164    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:51.905965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.905938    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:11:51.909398 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:51.909368    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e139db_8e50_4f7e_a08d_c998302b9f49.slice/crio-be3f2f19c6aa74c12e43cb4d12a59757aa519f25e5092bbd2a24743bb6eb0181 WatchSource:0}: Error finding container be3f2f19c6aa74c12e43cb4d12a59757aa519f25e5092bbd2a24743bb6eb0181: Status 404 returned error can't find the container with id be3f2f19c6aa74c12e43cb4d12a59757aa519f25e5092bbd2a24743bb6eb0181
Nov 03 19:11:51.938540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.938511    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617920f7-62ac-4c2b-892e-182c5aa34bc2" path="/var/lib/kubelet/pods/617920f7-62ac-4c2b-892e-182c5aa34bc2/volumes"
Nov 03 19:11:51.939079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.939059    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739f1cef-139e-4df6-b99f-52acabd6f48e" path="/var/lib/kubelet/pods/739f1cef-139e-4df6-b99f-52acabd6f48e/volumes"
Nov 03 19:11:51.939583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:51.939566    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72094ff-35cd-4ed4-b40a-5a50dbc79c26" path="/var/lib/kubelet/pods/e72094ff-35cd-4ed4-b40a-5a50dbc79c26/volumes"
Nov 03 19:11:52.058076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.058051    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:11:52.060910 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:52.060884    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce11898_cc94_4875_9d78_4877672a3b72.slice/crio-e366841ebb32e9d6858cb70a1ceb57ce160a2eda679bc750d12755278d4ebc27 WatchSource:0}: Error finding container e366841ebb32e9d6858cb70a1ceb57ce160a2eda679bc750d12755278d4ebc27: Status 404 returned error can't find the container with id e366841ebb32e9d6858cb70a1ceb57ce160a2eda679bc750d12755278d4ebc27
Nov 03 19:11:52.324742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.324705    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" event={"ID":"fce11898-cc94-4875-9d78-4877672a3b72","Type":"ContainerStarted","Data":"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"}
Nov 03 19:11:52.325209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.324769    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" event={"ID":"fce11898-cc94-4875-9d78-4877672a3b72","Type":"ContainerStarted","Data":"e366841ebb32e9d6858cb70a1ceb57ce160a2eda679bc750d12755278d4ebc27"}
Nov 03 19:11:52.325209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.324811    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:11:52.326490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.326465    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" event={"ID":"75e139db-8e50-4f7e-a08d-c998302b9f49","Type":"ContainerStarted","Data":"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"}
Nov 03 19:11:52.326612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.326506    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" event={"ID":"75e139db-8e50-4f7e-a08d-c998302b9f49","Type":"ContainerStarted","Data":"be3f2f19c6aa74c12e43cb4d12a59757aa519f25e5092bbd2a24743bb6eb0181"}
Nov 03 19:11:52.326612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.326526    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:11:52.435501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.435453    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" podStartSLOduration=1.43544047 podStartE2EDuration="1.43544047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:52.43294819 +0000 UTC m=+1610.710301772" watchObservedRunningTime="2026-11-03 19:11:52.43544047 +0000 UTC m=+1610.712794053"
Nov 03 19:11:52.545962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.545910    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" podStartSLOduration=1.545892722 podStartE2EDuration="1.545892722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:52.54119877 +0000 UTC m=+1610.818552367" watchObservedRunningTime="2026-11-03 19:11:52.545892722 +0000 UTC m=+1610.823246304"
Nov 03 19:11:52.634020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:52.633923    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:54.305401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.305369    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:54.547177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.547104    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:54.801573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.801537    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"]
Nov 03 19:11:54.806429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.806411    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:54.861370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.861340    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"]
Nov 03 19:11:54.899417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.899391    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:11:54.899615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.899483    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:11:54.899615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.899508    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:11:54.903679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.903658    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:11:54.940237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.940210    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:54.940375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.940280    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:54.940424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:54.940401    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7q9k\" (UniqueName: \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.041505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.041471    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.041680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.041565    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.041680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.041660    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7q9k\" (UniqueName: \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.042230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.042205    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.043849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.043825    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.099468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.099393    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7q9k\" (UniqueName: \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\") pod \"maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.122185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.122162    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:55.508525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:55.508498    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"]
Nov 03 19:11:55.511103 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:11:55.511075    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c162e6_8d06_4c40_8f04_55008a19bea4.slice/crio-17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8 WatchSource:0}: Error finding container 17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8: Status 404 returned error can't find the container with id 17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8
Nov 03 19:11:56.376095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:56.376064    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5" event={"ID":"f5c162e6-8d06-4c40-8f04-55008a19bea4","Type":"ContainerStarted","Data":"ea1e6a20d20077abe010be988249d050a5c342867915820722d247eeb8a014be"}
Nov 03 19:11:56.376220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:56.376159    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5" event={"ID":"f5c162e6-8d06-4c40-8f04-55008a19bea4","Type":"ContainerStarted","Data":"17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8"}
Nov 03 19:11:56.489501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:56.489457    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5" podStartSLOduration=2.489444087 podStartE2EDuration="2.489444087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:11:56.48331892 +0000 UTC m=+1614.760672503" watchObservedRunningTime="2026-11-03 19:11:56.489444087 +0000 UTC m=+1614.766797670"
Nov 03 19:11:57.387056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:57.387021    4944 generic.go:356] "Generic (PLEG): container finished" podID="f5c162e6-8d06-4c40-8f04-55008a19bea4" containerID="ea1e6a20d20077abe010be988249d050a5c342867915820722d247eeb8a014be" exitCode=0
Nov 03 19:11:57.387452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:57.387096    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5" event={"ID":"f5c162e6-8d06-4c40-8f04-55008a19bea4","Type":"ContainerDied","Data":"ea1e6a20d20077abe010be988249d050a5c342867915820722d247eeb8a014be"}
Nov 03 19:11:57.542469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:57.542428    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:11:58.557627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.557603    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:58.701074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.700993    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\") pod \"f5c162e6-8d06-4c40-8f04-55008a19bea4\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") "
Nov 03 19:11:58.701211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.701151    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\") pod \"f5c162e6-8d06-4c40-8f04-55008a19bea4\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") "
Nov 03 19:11:58.701428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.701400    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca" pod "f5c162e6-8d06-4c40-8f04-55008a19bea4" (UID: "f5c162e6-8d06-4c40-8f04-55008a19bea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:11:58.701470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.701431    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\" (UniqueName: \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\") pod \"f5c162e6-8d06-4c40-8f04-55008a19bea4\" (UID: \"f5c162e6-8d06-4c40-8f04-55008a19bea4\") "
Nov 03 19:11:58.702130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.702099    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:58.703558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.703526    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k" pod "f5c162e6-8d06-4c40-8f04-55008a19bea4" (UID: "f5c162e6-8d06-4c40-8f04-55008a19bea4"). InnerVolumeSpecName "kube-api-access-j7q9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:11:58.703692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.703576    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca" pod "f5c162e6-8d06-4c40-8f04-55008a19bea4" (UID: "f5c162e6-8d06-4c40-8f04-55008a19bea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:11:58.803418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.803377    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7q9k\" (UniqueName: \"kubernetes.io/projected/f5c162e6-8d06-4c40-8f04-55008a19bea4-kube-api-access-j7q9k\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:58.803418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:58.803409    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5c162e6-8d06-4c40-8f04-55008a19bea4-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:11:59.405420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:59.405385    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5" event={"ID":"f5c162e6-8d06-4c40-8f04-55008a19bea4","Type":"ContainerDied","Data":"17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8"}
Nov 03 19:11:59.405420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:59.405410    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"
Nov 03 19:11:59.405686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:11:59.405433    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17e8bd29983f742cfd30885ad4b8fbe43074595a9b619f3ad7206dcfa83bcee8"
Nov 03 19:12:01.049249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:01.049213    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"]
Nov 03 19:12:01.095470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:01.095437    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c7627b8-xw72l"]
Nov 03 19:12:01.836777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:01.836737    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:01.957231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:01.957190    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2794aa86-81a6-4339-a806-1f2ada4d9e02" path="/var/lib/kubelet/pods/2794aa86-81a6-4339-a806-1f2ada4d9e02/volumes"
Nov 03 19:12:03.340063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:03.340032    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:12:03.342744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:03.342704    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:12:04.052471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.052432    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:04.414897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.411147    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:04.763980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.747027    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:04.841829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.841775    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"]
Nov 03 19:12:04.856694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.856650    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-46ccd045-w2ggd"]
Nov 03 19:12:04.879970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.879921    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:12:04.901507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:04.901461    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy" containerID="cri-o://7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe" gracePeriod=30
Nov 03 19:12:05.000444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.000290    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:05.272655 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:12:05.272569    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:12:05.383782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.383746    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:05.865875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.865834    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:12:05.866861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.866835    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f5c162e6-8d06-4c40-8f04-55008a19bea4" containerName="revoke-keys"
Nov 03 19:12:05.866861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.866857    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c162e6-8d06-4c40-8f04-55008a19bea4" containerName="revoke-keys"
Nov 03 19:12:05.867044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.867004    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c162e6-8d06-4c40-8f04-55008a19bea4" containerName="revoke-keys"
Nov 03 19:12:05.875530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.875505    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.921369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.921333    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:12:05.941374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.941335    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3602559f-9cec-4cf0-adbc-48fcae42e824" path="/var/lib/kubelet/pods/3602559f-9cec-4cf0-adbc-48fcae42e824/volumes"
Nov 03 19:12:05.960531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.960501    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5b012841-openshift-default-dockercfg-776cr\""
Nov 03 19:12:05.980980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.980940    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981017    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981251    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981401    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981465    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981507    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8x6\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981542    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981582    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:05.981760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:05.981741    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.074113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.074054    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:06.083388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083356    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083481    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083525    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083563    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8x6\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083596    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083632    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083785    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083847    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083888    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083848    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.083998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.083910    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.084152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.084121    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.084531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.084493    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.085414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.085390    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.086700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.086628    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.087847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.087824    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.141108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.141036    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.141108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.141095    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8x6\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\") pod \"e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") " pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.195821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.195753    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:06.657717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.657679    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:12:06.661651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.661617    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:12:06.661769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.661685    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:12:06.661769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:06.661723    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:12:07.169147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.169115    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:07.325148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.325115    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:12:07.427688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.427601    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:12:07.569337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.569301    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerStarted","Data":"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"}
Nov 03 19:12:07.569509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.569353    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerStarted","Data":"44449c38f348e70d48923e969e7245dc87d03ea0dcd20fd94d9fb5478729874e"}
Nov 03 19:12:07.686304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.686201    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podStartSLOduration=2.6861845779999998 podStartE2EDuration="2.686184578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:07.68181774 +0000 UTC m=+1625.959171325" watchObservedRunningTime="2026-11-03 19:12:07.686184578 +0000 UTC m=+1625.963538163"
Nov 03 19:12:07.799017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:07.798981    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:08.072958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:08.072915    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:08.196862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:08.196781    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:08.202232 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:08.202203    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:08.578857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:08.578828    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:08.579845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:08.579824    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:12:09.625673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:09.625635    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:09.907840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:09.907573    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:10.159814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.159010    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:10.312496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.312473    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:12:10.385692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.385661    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.385859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.385842    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.385932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.385865    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.385999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.385936    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.385999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.385975    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.386113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386039    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.386113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386104    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.386215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386140    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.386215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386176    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\") pod \"9ad813bb-37fc-4eba-9c62-59799b12467f\" (UID: \"9ad813bb-37fc-4eba-9c62-59799b12467f\") "
Nov 03 19:12:10.386327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386245    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:10.386327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386274    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:10.386327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386310    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:10.386484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386434    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:10.386665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386639    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:10.386969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386949    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.387073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386972    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.387073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.386989    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ad813bb-37fc-4eba-9c62-59799b12467f-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.387073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.387001    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.387073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.387012    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.388372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.388339    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:12:10.388475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.388380    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:10.388475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.388406    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:12:10.388825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.388770    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn" pod "9ad813bb-37fc-4eba-9c62-59799b12467f" (UID: "9ad813bb-37fc-4eba-9c62-59799b12467f"). InnerVolumeSpecName "kube-api-access-5gjmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:10.489634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.488591    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.489634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.488630    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gjmn\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-kube-api-access-5gjmn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.489634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.488646    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.489634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.488660    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ad813bb-37fc-4eba-9c62-59799b12467f-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:10.607897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.607862    4944 generic.go:356] "Generic (PLEG): container finished" podID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerID="7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe" exitCode=0
Nov 03 19:12:10.608037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.607941    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"
Nov 03 19:12:10.608037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.607941    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" event={"ID":"9ad813bb-37fc-4eba-9c62-59799b12467f","Type":"ContainerDied","Data":"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"}
Nov 03 19:12:10.608286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.608046    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7" event={"ID":"9ad813bb-37fc-4eba-9c62-59799b12467f","Type":"ContainerDied","Data":"de379762d433eddf1fea7f0c1cbdda5f8f0211cc955daae00197eff5664bab1a"}
Nov 03 19:12:10.608286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.608081    4944 scope.go:122] "RemoveContainer" containerID="7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"
Nov 03 19:12:10.621159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.621143    4944 scope.go:122] "RemoveContainer" containerID="7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"
Nov 03 19:12:10.621418 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:12:10.621395    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe\": container with ID starting with 7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe not found: ID does not exist" containerID="7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"
Nov 03 19:12:10.621476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.621425    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe"} err="failed to get container status \"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe\": rpc error: code = NotFound desc = could not find container \"7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe\": container with ID starting with 7a5fd23fa4cb1884c9c8dfb7f9b8da5d73a1584986a21e910fbab96c9e4efefe not found: ID does not exist"
Nov 03 19:12:10.767134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.767052    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:12:10.772459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:10.772432    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-46ccd045-openshift-default-7db5d87767-c6kf7"]
Nov 03 19:12:11.051182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.051099    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"]
Nov 03 19:12:11.056156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.056127    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968e48c4-7xnqm"]
Nov 03 19:12:11.306089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.306010    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:11.580687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.580577    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:11.683514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.683476    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"]
Nov 03 19:12:11.689303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.689271    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e114ecf4-8p48p"]
Nov 03 19:12:11.824896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.824857    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:11.954976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.954903    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c67d676-185c-48e2-92c6-467e83964e5d" path="/var/lib/kubelet/pods/2c67d676-185c-48e2-92c6-467e83964e5d/volumes"
Nov 03 19:12:11.955864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.955780    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" path="/var/lib/kubelet/pods/9ad813bb-37fc-4eba-9c62-59799b12467f/volumes"
Nov 03 19:12:11.956571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:11.956550    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5250c46-ab57-415e-b9cd-f79ca7a4c7e0" path="/var/lib/kubelet/pods/d5250c46-ab57-415e-b9cd-f79ca7a4c7e0/volumes"
Nov 03 19:12:12.071353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.071312    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:12.325604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.325563    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" podUID="ffb3ffda-0fa6-468c-b82d-602474b89c97" containerName="payload-processing" containerID="cri-o://7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5" gracePeriod=25
Nov 03 19:12:12.428635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.428602    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" podUID="00977ece-f9ab-4316-9316-6e13de18bd0b" containerName="payload-pre-processing" containerID="cri-o://83434b4f8bcc323af135734549650c9d3bbccd36f12b9f90331f7caf8d73334a" gracePeriod=25
Nov 03 19:12:12.642923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.642895    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:12:12.644785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.644754    4944 generic.go:356] "Generic (PLEG): container finished" podID="00977ece-f9ab-4316-9316-6e13de18bd0b" containerID="83434b4f8bcc323af135734549650c9d3bbccd36f12b9f90331f7caf8d73334a" exitCode=0
Nov 03 19:12:12.644979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.644919    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" event={"ID":"00977ece-f9ab-4316-9316-6e13de18bd0b","Type":"ContainerDied","Data":"83434b4f8bcc323af135734549650c9d3bbccd36f12b9f90331f7caf8d73334a"}
Nov 03 19:12:12.646531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.646504    4944 generic.go:356] "Generic (PLEG): container finished" podID="ffb3ffda-0fa6-468c-b82d-602474b89c97" containerID="7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5" exitCode=0
Nov 03 19:12:12.646659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.646559    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"
Nov 03 19:12:12.646659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.646592    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" event={"ID":"ffb3ffda-0fa6-468c-b82d-602474b89c97","Type":"ContainerDied","Data":"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"}
Nov 03 19:12:12.646847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.646634    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw" event={"ID":"ffb3ffda-0fa6-468c-b82d-602474b89c97","Type":"ContainerDied","Data":"574604d351c3acbc2f4449acd8ba58cc69ad1f3d30f4a9f43553f8f25bfe449f"}
Nov 03 19:12:12.646847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.646693    4944 scope.go:122] "RemoveContainer" containerID="7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"
Nov 03 19:12:12.669027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.669004    4944 scope.go:122] "RemoveContainer" containerID="7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"
Nov 03 19:12:12.669410 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:12:12.669370    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5\": container with ID starting with 7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5 not found: ID does not exist" containerID="7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"
Nov 03 19:12:12.669501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.669403    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5"} err="failed to get container status \"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5\": rpc error: code = NotFound desc = could not find container \"7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5\": container with ID starting with 7b257a3b402be1bdbd383be9b3397b601b085555f1158bb4ff41b0209330eef5 not found: ID does not exist"
Nov 03 19:12:12.734877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.734852    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\" (UniqueName: \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\") pod \"ffb3ffda-0fa6-468c-b82d-602474b89c97\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") "
Nov 03 19:12:12.735152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.735133    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\") pod \"ffb3ffda-0fa6-468c-b82d-602474b89c97\" (UID: \"ffb3ffda-0fa6-468c-b82d-602474b89c97\") "
Nov 03 19:12:12.735704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.735669    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume" pod "ffb3ffda-0fa6-468c-b82d-602474b89c97" (UID: "ffb3ffda-0fa6-468c-b82d-602474b89c97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:12.737723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.737694    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj" pod "ffb3ffda-0fa6-468c-b82d-602474b89c97" (UID: "ffb3ffda-0fa6-468c-b82d-602474b89c97"). InnerVolumeSpecName "kube-api-access-tdlnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:12.739883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.739863    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:12:12.837230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.837192    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\" (UniqueName: \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\") pod \"00977ece-f9ab-4316-9316-6e13de18bd0b\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") "
Nov 03 19:12:12.837716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.837406    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\") pod \"00977ece-f9ab-4316-9316-6e13de18bd0b\" (UID: \"00977ece-f9ab-4316-9316-6e13de18bd0b\") "
Nov 03 19:12:12.837770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.837752    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume" pod "00977ece-f9ab-4316-9316-6e13de18bd0b" (UID: "00977ece-f9ab-4316-9316-6e13de18bd0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:12.838049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.838031    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlnj\" (UniqueName: \"kubernetes.io/projected/ffb3ffda-0fa6-468c-b82d-602474b89c97-kube-api-access-tdlnj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:12.838098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.838053    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00977ece-f9ab-4316-9316-6e13de18bd0b-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:12.838098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.838065    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffb3ffda-0fa6-468c-b82d-602474b89c97-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:12.839472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.839443    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm" pod "00977ece-f9ab-4316-9316-6e13de18bd0b" (UID: "00977ece-f9ab-4316-9316-6e13de18bd0b"). InnerVolumeSpecName "kube-api-access-mlrxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:12.874119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.874047    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:12.939370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:12.939339    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlrxm\" (UniqueName: \"kubernetes.io/projected/00977ece-f9ab-4316-9316-6e13de18bd0b-kube-api-access-mlrxm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:13.064365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.064330    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:12:13.069527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.069498    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7a8bab4a-df4ccddb8-htfsw"]
Nov 03 19:12:13.127459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.127375    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:13.659593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.659562    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"
Nov 03 19:12:13.659593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.659570    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww" event={"ID":"00977ece-f9ab-4316-9316-6e13de18bd0b","Type":"ContainerDied","Data":"05ea40b8dc3a88c100c77053e61163068f5629cdeebd0a41a95f9379ad67c5da"}
Nov 03 19:12:13.659872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.659666    4944 scope.go:122] "RemoveContainer" containerID="83434b4f8bcc323af135734549650c9d3bbccd36f12b9f90331f7caf8d73334a"
Nov 03 19:12:13.819704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.819669    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:12:13.824970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.824942    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7a8bab4a-6b6b7dc479-jg5ww"]
Nov 03 19:12:13.936510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.936433    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00977ece-f9ab-4316-9316-6e13de18bd0b" path="/var/lib/kubelet/pods/00977ece-f9ab-4316-9316-6e13de18bd0b/volumes"
Nov 03 19:12:13.937162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:13.937139    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb3ffda-0fa6-468c-b82d-602474b89c97" path="/var/lib/kubelet/pods/ffb3ffda-0fa6-468c-b82d-602474b89c97/volumes"
Nov 03 19:12:14.049520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:14.034325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:14.283011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:14.282969    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:16.432529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:16.432499    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:18.178426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:18.178396    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:18.433863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:18.433733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:18.647636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:18.647591    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"]
Nov 03 19:12:18.686647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:18.686564    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:18.693663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:18.693633    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c2fc8901-6lx9q"]
Nov 03 19:12:19.046920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:19.046886    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"]
Nov 03 19:12:19.068343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:19.068313    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb26413f-nzfzs"]
Nov 03 19:12:19.936374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:19.936336    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c80e97-a106-446a-9a38-c1ff68bf1c0a" path="/var/lib/kubelet/pods/b6c80e97-a106-446a-9a38-c1ff68bf1c0a/volumes"
Nov 03 19:12:19.936937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:19.936922    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9c7cd95-a084-4614-a208-7ca4965cba2b" path="/var/lib/kubelet/pods/c9c7cd95-a084-4614-a208-7ca4965cba2b/volumes"
Nov 03 19:12:20.326366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:20.326331    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:23.640284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:23.640243    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:23.889555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:23.889521    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:25.926208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:25.926172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:26.388510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:26.388475    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:27.069377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.069335    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"]
Nov 03 19:12:27.112122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.112081    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:27.120004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.119964    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ae8152d-5579q"]
Nov 03 19:12:27.506543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.506511    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:27.875340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.875259    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:27.936804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:27.936767    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2544645-038a-4c1d-9ce2-3b2bdd1624a1" path="/var/lib/kubelet/pods/b2544645-038a-4c1d-9ce2-3b2bdd1624a1/volumes"
Nov 03 19:12:29.713252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:29.713205    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:29.982411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:29.979903    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:30.145041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:30.145005    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"]
Nov 03 19:12:30.189746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:30.189700    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7a8bab4a-bgpg5"]
Nov 03 19:12:30.229198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:30.229163    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:30.662813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:30.656023    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:31.940514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:31.940470    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c162e6-8d06-4c40-8f04-55008a19bea4" path="/var/lib/kubelet/pods/f5c162e6-8d06-4c40-8f04-55008a19bea4/volumes"
Nov 03 19:12:32.750641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:32.750609    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:33.039902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:33.039779    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:34.058134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:34.058100    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"]
Nov 03 19:12:34.104005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:34.103968    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c9e7f23-cfrqh"]
Nov 03 19:12:35.225452 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:12:35.225416    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:12:35.628206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:35.628172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:35.937088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:35.937003    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4930c3-883b-4d35-b8b1-259113efe96d" path="/var/lib/kubelet/pods/ec4930c3-883b-4d35-b8b1-259113efe96d/volumes"
Nov 03 19:12:36.090073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.090038    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:36.357169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.357134    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:36.594151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.584602    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:12:36.595705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.595669    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ffb3ffda-0fa6-468c-b82d-602474b89c97" containerName="payload-processing"
Nov 03 19:12:36.595905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.595888    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb3ffda-0fa6-468c-b82d-602474b89c97" containerName="payload-processing"
Nov 03 19:12:36.596052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.596039    4944 cpu_manager.go:414] "containerMap: removing container" podUID="00977ece-f9ab-4316-9316-6e13de18bd0b" containerName="payload-pre-processing"
Nov 03 19:12:36.596173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.596161    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00977ece-f9ab-4316-9316-6e13de18bd0b" containerName="payload-pre-processing"
Nov 03 19:12:36.596307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.596295    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy"
Nov 03 19:12:36.596392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.596382    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy"
Nov 03 19:12:36.596641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.596627    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="00977ece-f9ab-4316-9316-6e13de18bd0b" containerName="payload-pre-processing"
Nov 03 19:12:36.628113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.628050    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad813bb-37fc-4eba-9c62-59799b12467f" containerName="istio-proxy"
Nov 03 19:12:36.628264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.628252    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb3ffda-0fa6-468c-b82d-602474b89c97" containerName="payload-processing"
Nov 03 19:12:36.653493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.653459    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.665275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.665230    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:12:36.780494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.780456    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:12:36.797668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.797634    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:36.827455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.827422    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:12:36.828312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.827772    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d86d1cd0\""
Nov 03 19:12:36.828493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.827952    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d86d1cd0-dockercfg-pfzr7\""
Nov 03 19:12:36.829368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.829302    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.829477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.829464    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7jc\" (UniqueName: \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.930710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.930613    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.930710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.930705    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:36.930960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.930816    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7jc\" (UniqueName: \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.930960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.930926    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qxzq\" (UniqueName: \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:36.931882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.931851    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:36.990701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:36.990667    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7jc\" (UniqueName: \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\") pod \"payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") " pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:37.032526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.032485    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.033668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.033560    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.033926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.033902    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qxzq\" (UniqueName: \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.075460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.075430    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:37.092146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.091978    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qxzq\" (UniqueName: \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\") pod \"payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.151717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.144058    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.497633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.497569    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:12:37.501051 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:12:37.501023    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4972333b_64e8_42cf_8420_c3caa5944b01.slice/crio-d8f976eec8092edb1211aecf07b7f2c3cb7398f2a07fa8ab001b639d3ecec67f WatchSource:0}: Error finding container d8f976eec8092edb1211aecf07b7f2c3cb7398f2a07fa8ab001b639d3ecec67f: Status 404 returned error can't find the container with id d8f976eec8092edb1211aecf07b7f2c3cb7398f2a07fa8ab001b639d3ecec67f
Nov 03 19:12:37.616781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.616752    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:12:37.621431 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:12:37.621399    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d6ef4f_07d3_4ab4_bac3_2cfa745470f6.slice/crio-37500783b59d7c36bed0ed479a2d0678d792e43dbc2a34b6db1eec834eb3fac7 WatchSource:0}: Error finding container 37500783b59d7c36bed0ed479a2d0678d792e43dbc2a34b6db1eec834eb3fac7: Status 404 returned error can't find the container with id 37500783b59d7c36bed0ed479a2d0678d792e43dbc2a34b6db1eec834eb3fac7
Nov 03 19:12:37.734829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.734770    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:37.942942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.942850    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" event={"ID":"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6","Type":"ContainerStarted","Data":"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"}
Nov 03 19:12:37.942942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.942909    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" event={"ID":"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6","Type":"ContainerStarted","Data":"37500783b59d7c36bed0ed479a2d0678d792e43dbc2a34b6db1eec834eb3fac7"}
Nov 03 19:12:37.943178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.943073    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:37.944742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.944714    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" event={"ID":"4972333b-64e8-42cf-8420-c3caa5944b01","Type":"ContainerStarted","Data":"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"}
Nov 03 19:12:37.944910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.944761    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" event={"ID":"4972333b-64e8-42cf-8420-c3caa5944b01","Type":"ContainerStarted","Data":"d8f976eec8092edb1211aecf07b7f2c3cb7398f2a07fa8ab001b639d3ecec67f"}
Nov 03 19:12:37.944910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:37.944829    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:38.058562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:38.058493    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" podStartSLOduration=2.058473123 podStartE2EDuration="2.058473123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:38.054125147 +0000 UTC m=+1656.331478734" watchObservedRunningTime="2026-11-03 19:12:38.058473123 +0000 UTC m=+1656.335826903"
Nov 03 19:12:38.171484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:38.171404    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" podStartSLOduration=2.171390488 podStartE2EDuration="2.171390488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:12:38.167200083 +0000 UTC m=+1656.444553667" watchObservedRunningTime="2026-11-03 19:12:38.171390488 +0000 UTC m=+1656.448744070"
Nov 03 19:12:38.226829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:38.226698    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:38.488606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:38.488518    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:38.778424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:38.778171    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:39.067343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:39.067268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:39.394782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:39.394689    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:39.662678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:39.662593    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:40.958374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:40.958242    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:45.662682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:45.662646    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:45.920400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:45.920319    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:46.603929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:46.603895    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:46.895528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:46.895413    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:48.009842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:48.009804    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:48.314030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:48.313935    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:48.959616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:48.959526    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:12:48.960033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:48.960000    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:12:50.141690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.141653    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"]
Nov 03 19:12:50.146114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.146095    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.243970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.243939    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:12:50.244140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.243936    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:12:50.244140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.243990    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:12:50.244140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.244123    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:12:50.249463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.249441    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"]
Nov 03 19:12:50.257252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.257228    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.257365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.257346    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.257605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.257586    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bd5\" (UniqueName: \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.358956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.358925    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bd5\" (UniqueName: \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.359141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.358974    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.359141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.359048    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.359621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.359593    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.361526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.361504    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.417303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.417240    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bd5\" (UniqueName: \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\") pod \"maas-api-revoke-sub-d7afd59d-scslm\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.461651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.461621    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:50.620049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.620017    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:50.841363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:50.841332    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"]
Nov 03 19:12:50.848156 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:12:50.848122    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b3d180_c4a9_4700_a8e6_6e98b947dd13.slice/crio-8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183 WatchSource:0}: Error finding container 8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183: Status 404 returned error can't find the container with id 8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183
Nov 03 19:12:51.081534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:51.081500    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm" event={"ID":"c7b3d180-c4a9-4700-a8e6-6e98b947dd13","Type":"ContainerStarted","Data":"8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183"}
Nov 03 19:12:52.091753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:52.091663    4944 generic.go:356] "Generic (PLEG): container finished" podID="c7b3d180-c4a9-4700-a8e6-6e98b947dd13" containerID="8cc6a2964453edce66eefc8cf00c5e91520d321eb26a2d483f3697c7d8e4169e" exitCode=0
Nov 03 19:12:52.092173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:52.091814    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm" event={"ID":"c7b3d180-c4a9-4700-a8e6-6e98b947dd13","Type":"ContainerDied","Data":"8cc6a2964453edce66eefc8cf00c5e91520d321eb26a2d483f3697c7d8e4169e"}
Nov 03 19:12:53.289204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.289177    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:53.411016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.410938    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\" (UniqueName: \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\") pod \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") "
Nov 03 19:12:53.411016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.411009    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\") pod \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") "
Nov 03 19:12:53.411247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.411075    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\") pod \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\" (UID: \"c7b3d180-c4a9-4700-a8e6-6e98b947dd13\") "
Nov 03 19:12:53.412310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.412274    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca" pod "c7b3d180-c4a9-4700-a8e6-6e98b947dd13" (UID: "c7b3d180-c4a9-4700-a8e6-6e98b947dd13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:12:53.413919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.413893    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca" pod "c7b3d180-c4a9-4700-a8e6-6e98b947dd13" (UID: "c7b3d180-c4a9-4700-a8e6-6e98b947dd13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:12:53.414525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.414504    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5" pod "c7b3d180-c4a9-4700-a8e6-6e98b947dd13" (UID: "c7b3d180-c4a9-4700-a8e6-6e98b947dd13"). InnerVolumeSpecName "kube-api-access-d9bd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:12:53.512587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.512556    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9bd5\" (UniqueName: \"kubernetes.io/projected/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-kube-api-access-d9bd5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:53.512587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.512586    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:53.512773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.512597    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3d180-c4a9-4700-a8e6-6e98b947dd13-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:12:53.515243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:53.515210    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:54.125405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:54.125309    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm" event={"ID":"c7b3d180-c4a9-4700-a8e6-6e98b947dd13","Type":"ContainerDied","Data":"8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183"}
Nov 03 19:12:54.125405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:54.125371    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8137a0fe0466afc8ddddac4a5a2da6b5fa86fc7012d70905236f21bd2df52183"
Nov 03 19:12:54.125689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:54.125669    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"
Nov 03 19:12:54.175778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:54.174184    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:54.445114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:54.445074    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:55.062287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:55.062254    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"]
Nov 03 19:12:55.103374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:55.103338    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21944872-zbvrv"]
Nov 03 19:12:55.399913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:55.399832    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:55.941234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:55.941194    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d18983-de37-422b-9928-78ff67798770" path="/var/lib/kubelet/pods/c1d18983-de37-422b-9928-78ff67798770/volumes"
Nov 03 19:12:56.049515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:56.049480    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"]
Nov 03 19:12:56.095066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:56.095038    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b116d3-cwss4"]
Nov 03 19:12:57.936268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:57.936232    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="558451a8-f2ef-435b-9699-04ebdec26e05" path="/var/lib/kubelet/pods/558451a8-f2ef-435b-9699-04ebdec26e05/volumes"
Nov 03 19:12:58.538540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:58.538502    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:58.835828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:58.835735    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:59.444431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:59.444393    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:12:59.752411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:12:59.752383    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:00.341426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:00.341347    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:00.591991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:00.591877    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:00.846288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:00.846165    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:01.106006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:01.105833    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:02.759144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:02.759114    4944 scope.go:122] "RemoveContainer" containerID="78cadb4705b1363fde34513d18501f8d093a9ed356e4f599a2ad12ccadcdee2b"
Nov 03 19:13:02.770269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:02.770249    4944 scope.go:122] "RemoveContainer" containerID="de69ac18f9b25ea9728eaf2a2bca8f03829d125d7690bc1b5e95fe4df0912e0f"
Nov 03 19:13:02.780493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:02.780470    4944 scope.go:122] "RemoveContainer" containerID="9fc8f84fbebbd9116f452347f96053872df1e1bcdc237bcc24fc24c12898dbee"
Nov 03 19:13:02.796190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:02.795915    4944 scope.go:122] "RemoveContainer" containerID="a4b59831dc22671e90b6f9b285c1f51b5f74e3a0a495deeb0876fe3d5107b158"
Nov 03 19:13:02.818340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:02.818321    4944 scope.go:122] "RemoveContainer" containerID="f6fc4fe678725207445867fa113edf5c838454b53cac5a9353a8e1186a5cdbb0"
Nov 03 19:13:03.698866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:03.695952    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:03.941038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:03.941003    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:04.010481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.010446    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:13:04.011462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.011439    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c7b3d180-c4a9-4700-a8e6-6e98b947dd13" containerName="revoke-keys"
Nov 03 19:13:04.011462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.011460    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b3d180-c4a9-4700-a8e6-6e98b947dd13" containerName="revoke-keys"
Nov 03 19:13:04.011635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.011589    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b3d180-c4a9-4700-a8e6-6e98b947dd13" containerName="revoke-keys"
Nov 03 19:13:04.017430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.017410    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.070419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.070384    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:13:04.160810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.152058    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.160810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.152290    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjhb7\" (UniqueName: \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.170655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.170615    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4702dd-dockercfg-dz4vs\""
Nov 03 19:13:04.171037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.171015    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4702dd\""
Nov 03 19:13:04.215957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.215923    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:13:04.251812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.237677    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:13:04.253969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.244158    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.254236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.253592    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjhb7\" (UniqueName: \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.255951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.254489    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.255951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.255537    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.312434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.312396    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjhb7\" (UniqueName: \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\") pod \"payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.333001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.332973    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:04.356165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.356132    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.356320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.356295    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lms\" (UniqueName: \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.482436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.481463    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.482638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.482592    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lms\" (UniqueName: \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.483101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.482458    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.541502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.541391    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lms\" (UniqueName: \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\") pod \"payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.581817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.580130    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:04.719276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.719245    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:13:04.722874 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:04.722841    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a3107f_9d54_470c_8587_f4bdef15f9bc.slice/crio-26a9f65a5a1d36861ea9690b0d6b0cfd62facec1e474bed286df989eb8060d63 WatchSource:0}: Error finding container 26a9f65a5a1d36861ea9690b0d6b0cfd62facec1e474bed286df989eb8060d63: Status 404 returned error can't find the container with id 26a9f65a5a1d36861ea9690b0d6b0cfd62facec1e474bed286df989eb8060d63
Nov 03 19:13:04.983229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:04.983203    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:13:04.986339 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:04.986306    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373578f5_bb60_4487_8f65_22f9292a2a79.slice/crio-7285f3e8df80fb0a6b67f8ad922811bf38d61d54af43bc1ea60180e08f6e0029 WatchSource:0}: Error finding container 7285f3e8df80fb0a6b67f8ad922811bf38d61d54af43bc1ea60180e08f6e0029: Status 404 returned error can't find the container with id 7285f3e8df80fb0a6b67f8ad922811bf38d61d54af43bc1ea60180e08f6e0029
Nov 03 19:13:05.226477 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:13:05.226438    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:13:05.265746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.265712    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" event={"ID":"53a3107f-9d54-470c-8587-f4bdef15f9bc","Type":"ContainerStarted","Data":"bd9aa025445ab7a64716cac1d632c6e43028a6650627833c3a2bc4258f7a7e0c"}
Nov 03 19:13:05.265950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.265770    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" event={"ID":"53a3107f-9d54-470c-8587-f4bdef15f9bc","Type":"ContainerStarted","Data":"26a9f65a5a1d36861ea9690b0d6b0cfd62facec1e474bed286df989eb8060d63"}
Nov 03 19:13:05.265950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.265813    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:05.267419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.267389    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" event={"ID":"373578f5-bb60-4487-8f65-22f9292a2a79","Type":"ContainerStarted","Data":"d9bc39daa315eb5a348761e37d5290933276312fd1dffb54c827b599b82991eb"}
Nov 03 19:13:05.267535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.267449    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:05.267535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.267476    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" event={"ID":"373578f5-bb60-4487-8f65-22f9292a2a79","Type":"ContainerStarted","Data":"7285f3e8df80fb0a6b67f8ad922811bf38d61d54af43bc1ea60180e08f6e0029"}
Nov 03 19:13:05.377644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.377530    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" podStartSLOduration=2.377498956 podStartE2EDuration="2.377498956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:05.373861985 +0000 UTC m=+1683.651215570" watchObservedRunningTime="2026-11-03 19:13:05.377498956 +0000 UTC m=+1683.654852577"
Nov 03 19:13:05.484487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:05.484434    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" podStartSLOduration=1.484419314 podStartE2EDuration="1.484419314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:05.482435798 +0000 UTC m=+1683.759789383" watchObservedRunningTime="2026-11-03 19:13:05.484419314 +0000 UTC m=+1683.761772897"
Nov 03 19:13:06.016199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:06.016106    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:06.921247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:06.921212    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:11.448813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:11.448299    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:11.993816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:11.992856    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": context deadline exceeded"
Nov 03 19:13:12.110000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.109938    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.148057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.148002    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 03 19:13:12.184814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.177412    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.189933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.189897    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:12.190147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.190123    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.342849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.342737    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.356812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.347880    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:12.356812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.347935    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.475933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.475888    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:13:12.476533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.476499    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.614865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.614377    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.651840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.651553    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.652039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.651898    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:13:12.659818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.653042    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:12.662150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:12.661864    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:15.075725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.075689    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"]
Nov 03 19:13:15.137631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.137588    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce701d90-9x9tv"]
Nov 03 19:13:15.681485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.681425    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:15.881474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.881443    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:13:15.881719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.881694    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy" containerID="cri-o://4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd" gracePeriod=30
Nov 03 19:13:15.939903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.939828    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce87f234-5f06-4f98-87db-e2114177567c" path="/var/lib/kubelet/pods/ce87f234-5f06-4f98-87db-e2114177567c/volumes"
Nov 03 19:13:15.940199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.940184    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:13:15.940616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:15.940577    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy" containerID="cri-o://e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb" gracePeriod=30
Nov 03 19:13:16.071481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:16.071445    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"]
Nov 03 19:13:16.076277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:16.076250    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-488a6e86-ccx9h"]
Nov 03 19:13:16.225556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:16.225480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:16.315877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:16.307858    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:13:16.315877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:16.314046    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:13:17.053699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:17.053660    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"]
Nov 03 19:13:17.061388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:17.061359    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce7399fc-q9g7d"]
Nov 03 19:13:17.825348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:17.825290    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:17.953213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:17.953175    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3097ba8e-57fe-47d0-bb93-7bc5daccee74" path="/var/lib/kubelet/pods/3097ba8e-57fe-47d0-bb93-7bc5daccee74/volumes"
Nov 03 19:13:17.953718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:17.953704    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6af61ef8-fa64-41e5-8e6a-f7c1c3157f54" path="/var/lib/kubelet/pods/6af61ef8-fa64-41e5-8e6a-f7c1c3157f54/volumes"
Nov 03 19:13:18.113180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:18.113072    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:19.051562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:19.051533    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"]
Nov 03 19:13:19.096680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:19.096640    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e013309-mpczx"]
Nov 03 19:13:19.941176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:19.941146    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a47c01-0678-439c-8a1c-b96a5828b158" path="/var/lib/kubelet/pods/b4a47c01-0678-439c-8a1c-b96a5828b158/volumes"
Nov 03 19:13:20.599286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.599248    4944 generic.go:356] "Generic (PLEG): container finished" podID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerID="cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d" exitCode=0
Nov 03 19:13:20.599688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.599311    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerDied","Data":"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"}
Nov 03 19:13:20.599753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.599737    4944 scope.go:122] "RemoveContainer" containerID="cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"
Nov 03 19:13:20.601383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.601346    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:20.601499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.601437    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:20.601499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:20.601486    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:21.119085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.119041    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:21.351365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.351335    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369069    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369128    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369242    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369312    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369320    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369379    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369426    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369518    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369555    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369641    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369693    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\") pod \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\" (UID: \"7f32810d-48e1-4428-a6bf-c4844a8bd71d\") "
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.369880    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.370321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370060    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.371223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370539    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.371223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370563    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.371223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370580    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.371223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370595    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.371223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.370859    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.373706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.373665    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.373985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.373949    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:21.374529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.374503    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:21.374767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.374749    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52" pod "7f32810d-48e1-4428-a6bf-c4844a8bd71d" (UID: "7f32810d-48e1-4428-a6bf-c4844a8bd71d"). InnerVolumeSpecName "kube-api-access-4zc52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:21.446266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.446133    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:13:21.471861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.471826    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.471888    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.471956    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472142    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472184    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472412    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.472437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472437    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472505    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472537    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.472644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472554    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.472851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472656    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.472851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.472725    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\") pod \"5e84a970-7ab9-472e-94f5-e512849cc6d7\" (UID: \"5e84a970-7ab9-472e-94f5-e512849cc6d7\") "
Nov 03 19:13:21.473445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473414    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zc52\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-kube-api-access-4zc52\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473439    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473452    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473467    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473481    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473494    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473507    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.473623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473519    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f32810d-48e1-4428-a6bf-c4844a8bd71d-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.474367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.473977    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:13:21.482566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.482052    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.482901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.482766    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:13:21.484429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.484399    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "kube-api-access-mkptm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:21.484725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.484697    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:13:21.484956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.484932    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token" pod "5e84a970-7ab9-472e-94f5-e512849cc6d7" (UID: "5e84a970-7ab9-472e-94f5-e512849cc6d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:13:21.574713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574677    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.574713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574711    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.574713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574721    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkptm\" (UniqueName: \"kubernetes.io/projected/5e84a970-7ab9-472e-94f5-e512849cc6d7-kube-api-access-mkptm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.574991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574732    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e84a970-7ab9-472e-94f5-e512849cc6d7-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.574991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574741    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.574991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.574750    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e84a970-7ab9-472e-94f5-e512849cc6d7-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:13:21.630868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.630185    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerStarted","Data":"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"}
Nov 03 19:13:21.633733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.633699    4944 generic.go:356] "Generic (PLEG): container finished" podID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerID="4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd" exitCode=0
Nov 03 19:13:21.634234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.634209    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" event={"ID":"7f32810d-48e1-4428-a6bf-c4844a8bd71d","Type":"ContainerDied","Data":"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"}
Nov 03 19:13:21.634397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.634382    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd" event={"ID":"7f32810d-48e1-4428-a6bf-c4844a8bd71d","Type":"ContainerDied","Data":"3ad6367d6231e547058706af8b7f690f8f44261c7ac9d73ca117e200a972eca7"}
Nov 03 19:13:21.634526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.634514    4944 scope.go:122] "RemoveContainer" containerID="4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"
Nov 03 19:13:21.638740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.638719    4944 generic.go:356] "Generic (PLEG): container finished" podID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerID="e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb" exitCode=0
Nov 03 19:13:21.638992 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.638974    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" event={"ID":"5e84a970-7ab9-472e-94f5-e512849cc6d7","Type":"ContainerDied","Data":"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"}
Nov 03 19:13:21.639126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.639110    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq" event={"ID":"5e84a970-7ab9-472e-94f5-e512849cc6d7","Type":"ContainerDied","Data":"ea65e4d1df3e3f4042f3f492779a6e849037f9c12d9912fe479e7d3fdb869dc5"}
Nov 03 19:13:21.642680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.642651    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"
Nov 03 19:13:21.647524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.643521    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"
Nov 03 19:13:21.673736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.673709    4944 scope.go:122] "RemoveContainer" containerID="4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"
Nov 03 19:13:21.674320 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:13:21.674294    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd\": container with ID starting with 4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd not found: ID does not exist" containerID="4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"
Nov 03 19:13:21.674527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.674504    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd"} err="failed to get container status \"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd\": rpc error: code = NotFound desc = could not find container \"4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd\": container with ID starting with 4ecb9783e11dc16480c0cb867065611bfc857c89711aed369cf4d07d554ec5fd not found: ID does not exist"
Nov 03 19:13:21.674631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.674619    4944 scope.go:122] "RemoveContainer" containerID="e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"
Nov 03 19:13:21.690129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.690109    4944 scope.go:122] "RemoveContainer" containerID="e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"
Nov 03 19:13:21.690427 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:13:21.690393    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb\": container with ID starting with e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb not found: ID does not exist" containerID="e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"
Nov 03 19:13:21.690503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.690438    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb"} err="failed to get container status \"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb\": rpc error: code = NotFound desc = could not find container \"e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb\": container with ID starting with e014129920b814ca35f7f4a9401935fa178178e47f708482a726add1080f80fb not found: ID does not exist"
Nov 03 19:13:21.863321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.863278    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:13:21.869461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.869433    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-754dd"]
Nov 03 19:13:21.943669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.943627    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" path="/var/lib/kubelet/pods/7f32810d-48e1-4428-a6bf-c4844a8bd71d/volumes"
Nov 03 19:13:21.979095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.979064    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:13:21.983963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:21.983932    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-tkwwq"]
Nov 03 19:13:22.032154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:22.032026    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:13:22.033872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:22.033837    4944 patch_prober.go:29] interesting pod/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 19:13:22.034042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:22.033905    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 19:13:22.229335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:22.229217    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:22.494996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:22.494965    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:23.024814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:23.019193    4944 patch_prober.go:29] interesting pod/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 19:13:23.024814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:23.019258    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 19:13:23.176073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:23.176043    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:13:23.943655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:23.943613    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" path="/var/lib/kubelet/pods/5e84a970-7ab9-472e-94f5-e512849cc6d7/volumes"
Nov 03 19:13:24.026672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:24.026637    4944 patch_prober.go:29] interesting pod/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 19:13:24.027260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:24.027232    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 19:13:25.031845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.023932    4944 patch_prober.go:29] interesting pod/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 19:13:25.031845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.024009    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 19:13:25.533135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.532639    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:25.832934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.832396    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:25.837106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.837071    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:13:25.838531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.838403    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy"
Nov 03 19:13:25.838695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.838677    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy"
Nov 03 19:13:25.838898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.838882    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy"
Nov 03 19:13:25.839009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.838996    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy"
Nov 03 19:13:25.839293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.839274    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f32810d-48e1-4428-a6bf-c4844a8bd71d" containerName="istio-proxy"
Nov 03 19:13:25.839409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.839396    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e84a970-7ab9-472e-94f5-e512849cc6d7" containerName="istio-proxy"
Nov 03 19:13:25.847445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.844729    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:25.895879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.895845    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:13:25.945376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:25.945340    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-08baeb\""
Nov 03 19:13:26.076008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.013166    4944 patch_prober.go:29] interesting pod/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body=
Nov 03 19:13:26.076591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.076562    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused"
Nov 03 19:13:26.077158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.074020    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.077392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.077374    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4v6\" (UniqueName: \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.077712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.040631    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-08baeb-dockercfg-h9smm\""
Nov 03 19:13:26.103741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.103629    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:13:26.147509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.146400    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:13:26.147509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.146470    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"]
Nov 03 19:13:26.147509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.146513    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd98f58-24pdw"]
Nov 03 19:13:26.147509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.146589    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.195269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.195225    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.195841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.195808    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9z5\" (UniqueName: \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.196079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.196059    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.196278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.196250    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4v6\" (UniqueName: \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.197045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.197014    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.288378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.274663    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4v6\" (UniqueName: \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\") pod \"payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.305817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.303672    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.305817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.303833    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9z5\" (UniqueName: \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.306035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.306015    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.387865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.378109    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9z5\" (UniqueName: \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\") pod \"payload-processing-e2e-api-a-08baeb-b44658846-k5qm7\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") " pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.387865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.380294    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:26.469313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.469219    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:26.498353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.498320    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:26.683061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.682321    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:26.921054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:26.920977    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:13:26.925646 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:26.925612    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15f1e3c9_b2d0_4e70_a62e_2da984c783e5.slice/crio-dfe5699057b3a0a8d16c7fe7e257af16a2d3fd9c9ccfe3e548015bdc65081fdc WatchSource:0}: Error finding container dfe5699057b3a0a8d16c7fe7e257af16a2d3fd9c9ccfe3e548015bdc65081fdc: Status 404 returned error can't find the container with id dfe5699057b3a0a8d16c7fe7e257af16a2d3fd9c9ccfe3e548015bdc65081fdc
Nov 03 19:13:27.028086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.028057    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:13:27.031892 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:27.031855    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd829026c_b5c2_427e_b43a_278701572fba.slice/crio-5320591462e5928620a4a55f5a26bb9282dac7fcb8b055997d7cb2266da9e70b WatchSource:0}: Error finding container 5320591462e5928620a4a55f5a26bb9282dac7fcb8b055997d7cb2266da9e70b: Status 404 returned error can't find the container with id 5320591462e5928620a4a55f5a26bb9282dac7fcb8b055997d7cb2266da9e70b
Nov 03 19:13:27.142428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.142386    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:27.221029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.220936    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:13:27.222857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.222827    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:13:27.585666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.585626    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:27.717603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.717569    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" event={"ID":"d829026c-b5c2-427e-b43a-278701572fba","Type":"ContainerStarted","Data":"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"}
Nov 03 19:13:27.717833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.717673    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:27.717833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.717703    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" event={"ID":"d829026c-b5c2-427e-b43a-278701572fba","Type":"ContainerStarted","Data":"5320591462e5928620a4a55f5a26bb9282dac7fcb8b055997d7cb2266da9e70b"}
Nov 03 19:13:27.719452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.719425    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" event={"ID":"15f1e3c9-b2d0-4e70-a62e-2da984c783e5","Type":"ContainerStarted","Data":"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"}
Nov 03 19:13:27.719580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.719476    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" event={"ID":"15f1e3c9-b2d0-4e70-a62e-2da984c783e5","Type":"ContainerStarted","Data":"dfe5699057b3a0a8d16c7fe7e257af16a2d3fd9c9ccfe3e548015bdc65081fdc"}
Nov 03 19:13:27.719580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.719505    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:27.834583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.834527    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" podStartSLOduration=2.834512637 podStartE2EDuration="2.834512637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:27.829314853 +0000 UTC m=+1706.106668439" watchObservedRunningTime="2026-11-03 19:13:27.834512637 +0000 UTC m=+1706.111866238"
Nov 03 19:13:27.936367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.936292    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a404563-5183-452e-8989-9f5ffdff1309" path="/var/lib/kubelet/pods/0a404563-5183-452e-8989-9f5ffdff1309/volumes"
Nov 03 19:13:27.944828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:27.942697    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" podStartSLOduration=2.942678598 podStartE2EDuration="2.942678598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:27.93697825 +0000 UTC m=+1706.214331835" watchObservedRunningTime="2026-11-03 19:13:27.942678598 +0000 UTC m=+1706.220032182"
Nov 03 19:13:28.034286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.034236    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:28.557813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.532698    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:13:28.574928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.571667    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.701821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.698094    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-81a044-openshift-default-dockercfg-mm8ps\""
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.710843    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcts\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.710970    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711017    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711073    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711307    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711403    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711493    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711548    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.734635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.711582    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.738594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.738465    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:28.760929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.754308    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:13:28.812991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.812901    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.812994    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813033    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813069    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813178    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcts\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813280    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813318    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813368    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813445    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.813777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.813538    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.814493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.814035    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.814493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.814107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.814493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.814199    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.814723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.814691    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.815942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.815918    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.816260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.816240    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.870592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.870554    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.870994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.870973    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcts\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\") pod \"e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") " pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:28.946834 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:28.946762    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:29.056388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.054187    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"]
Nov 03 19:13:29.062645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.062266    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbcffd8a-8krx8"]
Nov 03 19:13:29.121973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.121829    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:29.406314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.406286    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:13:29.409635 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:29.409608    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d530c57_094d_4782_86d5_db5f87d9087f.slice/crio-11060c50fe303cc0d5a572a11feb50c2f97b95f2e4c229922e68bb63e8b0d8d2 WatchSource:0}: Error finding container 11060c50fe303cc0d5a572a11feb50c2f97b95f2e4c229922e68bb63e8b0d8d2: Status 404 returned error can't find the container with id 11060c50fe303cc0d5a572a11feb50c2f97b95f2e4c229922e68bb63e8b0d8d2
Nov 03 19:13:29.412160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.412127    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:29.412270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.412206    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:29.412270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.412249    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:29.806528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.806483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerStarted","Data":"19adb32c8dcc29686b815ff94f027dc5b345460b097f29ac5b89930b18687f99"}
Nov 03 19:13:29.807039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.806556    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerStarted","Data":"11060c50fe303cc0d5a572a11feb50c2f97b95f2e4c229922e68bb63e8b0d8d2"}
Nov 03 19:13:29.930099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.930044    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podStartSLOduration=1.930028798 podStartE2EDuration="1.930028798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:29.924196549 +0000 UTC m=+1708.201550131" watchObservedRunningTime="2026-11-03 19:13:29.930028798 +0000 UTC m=+1708.207382384"
Nov 03 19:13:29.937340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.937307    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e677ee-8b43-4150-b193-6e7d98b53bb2" path="/var/lib/kubelet/pods/a3e677ee-8b43-4150-b193-6e7d98b53bb2/volumes"
Nov 03 19:13:29.947413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:29.947395    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:30.956099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:30.955855    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:31.824747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:31.824714    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:31.825689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:31.825669    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:13:31.979999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:31.979955    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:33.396904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:33.396865    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:33.646718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:33.646662    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:35.226136 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:13:35.226100    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:13:36.912141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:36.912110    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:38.792397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:38.792368    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:13:38.792920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:38.792447    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:13:40.046939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:40.046900    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"]
Nov 03 19:13:40.092839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:40.092783    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23484ec6-wnbzw"]
Nov 03 19:13:41.263472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.263431    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:41.509525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.509484    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:41.565421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.565347    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:13:41.570072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.570052    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.627962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.627906    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:13:41.628766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.628734    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.628953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.628867    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmb5m\" (UniqueName: \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.683481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.683383    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-81a044\""
Nov 03 19:13:41.683970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.683945    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-81a044-dockercfg-tsf7p\""
Nov 03 19:13:41.751385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.751348    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:13:41.756283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.756237    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.756448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.756316    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmb5m\" (UniqueName: \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.757603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.757532    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.758689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.758659    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:41.797012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.796975    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:13:41.856727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.856602    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmb5m\" (UniqueName: \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\") pod \"payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.858189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.858155    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqsv\" (UniqueName: \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:41.858570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.858541    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:41.886321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.886289    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:41.943543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.943496    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880ab551-8781-47f4-a958-f9dbd5316d53" path="/var/lib/kubelet/pods/880ab551-8781-47f4-a958-f9dbd5316d53/volumes"
Nov 03 19:13:41.960030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.959999    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqsv\" (UniqueName: \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:41.960228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.960188    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:41.961048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:41.961008    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:42.096839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.094335    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:42.104053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.103032    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqsv\" (UniqueName: \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\") pod \"payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:42.297402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.297375    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:13:42.301260 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:42.301220    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ac16fa_1889_4ee2_954f_3a8a89c158a6.slice/crio-54f89ed4b5248275c2e329c63ebb8b39da2e50d7b831ac525df6fa759ef964ca WatchSource:0}: Error finding container 54f89ed4b5248275c2e329c63ebb8b39da2e50d7b831ac525df6fa759ef964ca: Status 404 returned error can't find the container with id 54f89ed4b5248275c2e329c63ebb8b39da2e50d7b831ac525df6fa759ef964ca
Nov 03 19:13:42.377806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.377763    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:42.761029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.760980    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:13:42.764435 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:13:42.764407    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c927f8_0b3d_4d64_9a7b_5b4172be113d.slice/crio-c5a93aadb390e0c72a8c295929f4613d6b0fb0bbdefb9024c0d25ae583afe96a WatchSource:0}: Error finding container c5a93aadb390e0c72a8c295929f4613d6b0fb0bbdefb9024c0d25ae583afe96a: Status 404 returned error can't find the container with id c5a93aadb390e0c72a8c295929f4613d6b0fb0bbdefb9024c0d25ae583afe96a
Nov 03 19:13:42.928523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.928487    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" event={"ID":"73c927f8-0b3d-4d64-9a7b-5b4172be113d","Type":"ContainerStarted","Data":"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"}
Nov 03 19:13:42.928711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.928541    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" event={"ID":"73c927f8-0b3d-4d64-9a7b-5b4172be113d","Type":"ContainerStarted","Data":"c5a93aadb390e0c72a8c295929f4613d6b0fb0bbdefb9024c0d25ae583afe96a"}
Nov 03 19:13:42.928711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.928581    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:42.930430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.930407    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" event={"ID":"71ac16fa-1889-4ee2-954f-3a8a89c158a6","Type":"ContainerStarted","Data":"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"}
Nov 03 19:13:42.930516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.930453    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" event={"ID":"71ac16fa-1889-4ee2-954f-3a8a89c158a6","Type":"ContainerStarted","Data":"54f89ed4b5248275c2e329c63ebb8b39da2e50d7b831ac525df6fa759ef964ca"}
Nov 03 19:13:42.930516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:42.930484    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:43.040290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:43.040182    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" podStartSLOduration=2.040167997 podStartE2EDuration="2.040167997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:43.036297567 +0000 UTC m=+1721.313651150" watchObservedRunningTime="2026-11-03 19:13:43.040167997 +0000 UTC m=+1721.317521580"
Nov 03 19:13:43.146928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:43.146875    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" podStartSLOduration=2.146860303 podStartE2EDuration="2.146860303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:13:43.143784273 +0000 UTC m=+1721.421137857" watchObservedRunningTime="2026-11-03 19:13:43.146860303 +0000 UTC m=+1721.424213886"
Nov 03 19:13:46.995762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:46.995726    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:48.081606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:48.081572    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"]
Nov 03 19:13:48.118829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:48.118760    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c62ba4d-vwqd6"]
Nov 03 19:13:50.003034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:50.002996    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b103112-1657-4877-b0bb-d76e9c68304a" path="/var/lib/kubelet/pods/1b103112-1657-4877-b0bb-d76e9c68304a/volumes"
Nov 03 19:13:50.079332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:50.079266    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"]
Nov 03 19:13:50.149820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:50.132726    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:13:50.175824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:50.154247    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-601b1566-s7ghx"]
Nov 03 19:13:51.895290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:51.895251    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:52.001516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:52.001464    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbac2237-5350-4959-8eec-71ad90cf13fc" path="/var/lib/kubelet/pods/dbac2237-5350-4959-8eec-71ad90cf13fc/volumes"
Nov 03 19:13:53.047463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:53.047425    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"]
Nov 03 19:13:53.093656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:53.093621    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f1654be-tzr9s"]
Nov 03 19:13:53.942416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:53.942381    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4be84a6c-e034-411a-9f48-a57865297246" path="/var/lib/kubelet/pods/4be84a6c-e034-411a-9f48-a57865297246/volumes"
Nov 03 19:13:53.945645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:53.945621    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:13:53.945780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:53.945684    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:13:56.046776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:56.046742    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"]
Nov 03 19:13:56.092815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:56.092769    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a15e442-mwbbb"]
Nov 03 19:13:56.461988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:56.461865    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:56.706143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:56.706100    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:56.954211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:56.954178    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:13:57.940904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:57.940862    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf" path="/var/lib/kubelet/pods/6c33d4f0-bc77-4cd0-a20b-020b8a93cbcf/volumes"
Nov 03 19:13:58.046922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:58.046888    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"]
Nov 03 19:13:58.093254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:58.093214    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383453ed-2qn4g"]
Nov 03 19:13:59.326623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.326589    4944 generic.go:356] "Generic (PLEG): container finished" podID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerID="e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c" exitCode=0
Nov 03 19:13:59.327097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.326641    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerDied","Data":"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"}
Nov 03 19:13:59.327153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.327139    4944 scope.go:122] "RemoveContainer" containerID="e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"
Nov 03 19:13:59.328865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.328839    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:59.328941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.328906    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:59.328978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.328939    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:13:59.936300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:13:59.936267    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e91fce8b-b8e5-4080-9c4a-0f21aaf7f384" path="/var/lib/kubelet/pods/e91fce8b-b8e5-4080-9c4a-0f21aaf7f384/volumes"
Nov 03 19:14:00.053317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:00.053278    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"]
Nov 03 19:14:00.058522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:00.058494    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cb64e58-rlbdf"]
Nov 03 19:14:00.350499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:00.350444    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerStarted","Data":"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"}
Nov 03 19:14:01.038298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.038253    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:01.049953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.049926    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"]
Nov 03 19:14:01.095694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.095667    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac9570e-bqp7g"]
Nov 03 19:14:01.196448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.196416    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:14:01.196610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.196557    4944 patch_prober.go:29] interesting pod/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 19:14:01.196610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.196595    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 19:14:01.346781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.346686    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:01.943209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.943176    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58aee447-e413-41fe-9bac-139641ff7aa8" path="/var/lib/kubelet/pods/58aee447-e413-41fe-9bac-139641ff7aa8/volumes"
Nov 03 19:14:01.943642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:01.943626    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798e000f-5bf6-44c0-9750-8ec25e46b3fa" path="/var/lib/kubelet/pods/798e000f-5bf6-44c0-9750-8ec25e46b3fa/volumes"
Nov 03 19:14:02.142433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.142382    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:02.249887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.249845    4944 patch_prober.go:29] interesting pod/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 19:14:02.255170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.255124    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 19:14:02.403078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.403049    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:02.932248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.932212    4944 scope.go:122] "RemoveContainer" containerID="5ea69b4a5f454614430e8d325d640556b39d91043a9b4ed6ce2e20f3a7f3d9e9"
Nov 03 19:14:02.944256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.944235    4944 scope.go:122] "RemoveContainer" containerID="037114c9ca2647b8c41198065232b9e690774c21be7f64e99fc5b467368b8cc0"
Nov 03 19:14:02.954527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.954509    4944 scope.go:122] "RemoveContainer" containerID="05fde70626bd0d4e12d5eab764a19e671966e738e309db2913e8f2c872a9b90c"
Nov 03 19:14:02.966483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.966460    4944 scope.go:122] "RemoveContainer" containerID="c9cb07c27131ca38b30322138b856ad2c34f2a24ebe2f0be6d74d029d4431aab"
Nov 03 19:14:02.975962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.975944    4944 scope.go:122] "RemoveContainer" containerID="ecacfa93ff1d0023283a83f90186ad147b8f65165d28bb9e5e5852bab74ffc83"
Nov 03 19:14:02.986467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:02.986411    4944 scope.go:122] "RemoveContainer" containerID="830a8ab7676c5e1123652e897a9757fd37f77f4b7e9d6830622ae530fbf6d194"
Nov 03 19:14:03.205882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:03.202757    4944 patch_prober.go:29] interesting pod/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 19:14:03.205882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:03.202892    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 19:14:04.048290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:04.048256    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"]
Nov 03 19:14:04.095287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:04.095249    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8d6f1cc-ggjb5"]
Nov 03 19:14:04.196493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:04.196455    4944 patch_prober.go:29] interesting pod/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 19:14:04.196662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:04.196518    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 19:14:05.197096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:05.197060    4944 patch_prober.go:29] interesting pod/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 03 19:14:05.197585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:05.197126    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 03 19:14:05.252081 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:14:05.251979    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:14:05.945811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:05.945760    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c29b35e-acdc-43a8-b34a-e623eb279238" path="/var/lib/kubelet/pods/5c29b35e-acdc-43a8-b34a-e623eb279238/volumes"
Nov 03 19:14:06.211754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:06.211682    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:14:06.212165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:06.211985    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:14:06.212858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:06.212842    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:14:06.496882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:06.496714    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:06.801843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:06.801748    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:07.122395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.122297    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"]
Nov 03 19:14:07.123115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.123085    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:07.129426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.129407    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.178201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.178174    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:14:07.178520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.178495    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:14:07.178520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.178224    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:14:07.179227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.178351    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:14:07.181842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.181818    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"]
Nov 03 19:14:07.182668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.182642    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.182856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.182832    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.183086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.183067    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2jd\" (UniqueName: \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.284137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.284104    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2jd\" (UniqueName: \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.284534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.284203    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.284534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.284256    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.285095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.285061    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.287301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.287265    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.342804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.342764    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2jd\" (UniqueName: \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\") pod \"maas-api-revoke-keys-e2e-derive-5b012841-cdjd7\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.445557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.445474    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:07.848952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:07.848918    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"]
Nov 03 19:14:07.853098 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:14:07.853070    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20393455_57b1_4c20_ac21_682b66815215.slice/crio-47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5 WatchSource:0}: Error finding container 47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5: Status 404 returned error can't find the container with id 47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5
Nov 03 19:14:08.458178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:08.458089    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7" event={"ID":"20393455-57b1-4c20-ac21-682b66815215","Type":"ContainerStarted","Data":"047398e3964510ed24437763825c5b7713b35b55e3db2ba7f7d30111be59dfd9"}
Nov 03 19:14:08.458178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:08.458150    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7" event={"ID":"20393455-57b1-4c20-ac21-682b66815215","Type":"ContainerStarted","Data":"47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5"}
Nov 03 19:14:08.578699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:08.578636    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7" podStartSLOduration=1.578617959 podStartE2EDuration="1.578617959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:08.566074117 +0000 UTC m=+1746.843427701" watchObservedRunningTime="2026-11-03 19:14:08.578617959 +0000 UTC m=+1746.855971544"
Nov 03 19:14:09.468389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:09.468351    4944 generic.go:356] "Generic (PLEG): container finished" podID="20393455-57b1-4c20-ac21-682b66815215" containerID="047398e3964510ed24437763825c5b7713b35b55e3db2ba7f7d30111be59dfd9" exitCode=0
Nov 03 19:14:09.468857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:09.468441    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7" event={"ID":"20393455-57b1-4c20-ac21-682b66815215","Type":"ContainerDied","Data":"047398e3964510ed24437763825c5b7713b35b55e3db2ba7f7d30111be59dfd9"}
Nov 03 19:14:10.157813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.150251    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:10.396159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.396131    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:10.429804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.429701    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:14:10.433840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.433784    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.491819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.491087    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:14:10.553813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.551106    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bbccf3\""
Nov 03 19:14:10.553813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.551260    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bbccf3-dockercfg-ns7mj\""
Nov 03 19:14:10.633840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.630379    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:14:10.712072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.711991    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.715908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.715722    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:14:10.760061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.759659    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.760061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.759739    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6r4q\" (UniqueName: \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.862195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.862142    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqln\" (UniqueName: \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.862369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.862290    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.862369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.862364    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.862478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.862459    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6r4q\" (UniqueName: \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.863107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.863082    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.876526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.876504    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:10.919976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.919946    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6r4q\" (UniqueName: \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\") pod \"payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.963874    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\") pod \"20393455-57b1-4c20-ac21-682b66815215\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") "
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.965147    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca" pod "20393455-57b1-4c20-ac21-682b66815215" (UID: "20393455-57b1-4c20-ac21-682b66815215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.964054    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\") pod \"20393455-57b1-4c20-ac21-682b66815215\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") "
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.965958    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\" (UniqueName: \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\") pod \"20393455-57b1-4c20-ac21-682b66815215\" (UID: \"20393455-57b1-4c20-ac21-682b66815215\") "
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.966369    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqln\" (UniqueName: \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.966457    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.966924    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20393455-57b1-4c20-ac21-682b66815215-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.967819    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.977937    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd" pod "20393455-57b1-4c20-ac21-682b66815215" (UID: "20393455-57b1-4c20-ac21-682b66815215"). InnerVolumeSpecName "kube-api-access-pk2jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:14:10.979491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:10.978914    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca" pod "20393455-57b1-4c20-ac21-682b66815215" (UID: "20393455-57b1-4c20-ac21-682b66815215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:14:11.032324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.032286    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqln\" (UniqueName: \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\") pod \"payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:11.038449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.038418    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:11.048664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.048637    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:11.068961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.068935    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2jd\" (UniqueName: \"kubernetes.io/projected/20393455-57b1-4c20-ac21-682b66815215-kube-api-access-pk2jd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:11.069103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.068966    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20393455-57b1-4c20-ac21-682b66815215-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:11.470423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.470390    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:14:11.474205 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:14:11.474180    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfcab9c7_614e_479e_8793_ba2d1d9b6427.slice/crio-6f576247c181027897cff5fd6f5bd5511659293b6cdedc89b0362d31c4f647ec WatchSource:0}: Error finding container 6f576247c181027897cff5fd6f5bd5511659293b6cdedc89b0362d31c4f647ec: Status 404 returned error can't find the container with id 6f576247c181027897cff5fd6f5bd5511659293b6cdedc89b0362d31c4f647ec
Nov 03 19:14:11.490769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.490737    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"
Nov 03 19:14:11.490908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.490736    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7" event={"ID":"20393455-57b1-4c20-ac21-682b66815215","Type":"ContainerDied","Data":"47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5"}
Nov 03 19:14:11.490908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.490871    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47701b07b18c8c66b43570931a703ba4a219a7d425bac5a4162f09c7e7c7d7b5"
Nov 03 19:14:11.492019 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.491998    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" event={"ID":"bfcab9c7-614e-479e-8793-ba2d1d9b6427","Type":"ContainerStarted","Data":"6f576247c181027897cff5fd6f5bd5511659293b6cdedc89b0362d31c4f647ec"}
Nov 03 19:14:11.520602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:11.520573    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:14:11.527238 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:14:11.527209    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634d6fee_d95b_443d_8ee7_25035c2fc0ce.slice/crio-c8467a5394318b534782e6b533ed2437ec6def65a3950961200666e2d81d74ad WatchSource:0}: Error finding container c8467a5394318b534782e6b533ed2437ec6def65a3950961200666e2d81d74ad: Status 404 returned error can't find the container with id c8467a5394318b534782e6b533ed2437ec6def65a3950961200666e2d81d74ad
Nov 03 19:14:12.008809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.008739    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:12.503316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.503283    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" event={"ID":"634d6fee-d95b-443d-8ee7-25035c2fc0ce","Type":"ContainerStarted","Data":"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"}
Nov 03 19:14:12.503724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.503345    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:12.503724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.503370    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" event={"ID":"634d6fee-d95b-443d-8ee7-25035c2fc0ce","Type":"ContainerStarted","Data":"c8467a5394318b534782e6b533ed2437ec6def65a3950961200666e2d81d74ad"}
Nov 03 19:14:12.505133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.505102    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" event={"ID":"bfcab9c7-614e-479e-8793-ba2d1d9b6427","Type":"ContainerStarted","Data":"502e0a391144e0864abfbc46ed4858f4ac9d3d48a4ce5ff619dcc0f4c004c300"}
Nov 03 19:14:12.505233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.505162    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:12.612044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.611984    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" podStartSLOduration=2.6119717590000002 podStartE2EDuration="2.611971759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:12.610996993 +0000 UTC m=+1750.888350602" watchObservedRunningTime="2026-11-03 19:14:12.611971759 +0000 UTC m=+1750.889325342"
Nov 03 19:14:12.734778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:12.734694    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" podStartSLOduration=2.734655425 podStartE2EDuration="2.734655425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:14:12.718923072 +0000 UTC m=+1750.996276681" watchObservedRunningTime="2026-11-03 19:14:12.734655425 +0000 UTC m=+1751.012009010"
Nov 03 19:14:14.049189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:14.049152    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"]
Nov 03 19:14:14.095468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:14.095435    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e2b203a-kwnjw"]
Nov 03 19:14:15.945179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:15.945142    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="698a7136-f5f3-4d97-944f-0263aa79be28" path="/var/lib/kubelet/pods/698a7136-f5f3-4d97-944f-0263aa79be28/volumes"
Nov 03 19:14:16.048249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:16.048216    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"]
Nov 03 19:14:16.092122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:16.092088    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62723de0-7clnb"]
Nov 03 19:14:17.105759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:17.105700    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:17.937005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:17.936971    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7598aedf-d8a8-4122-b232-cbc9f5a1018d" path="/var/lib/kubelet/pods/7598aedf-d8a8-4122-b232-cbc9f5a1018d/volumes"
Nov 03 19:14:19.415028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:19.414987    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:19.660617 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:19.660577    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:21.804907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:21.804872    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:22.057262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:22.057182    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:22.300007 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:22.299974    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:23.516491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:23.516458    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:14:23.516983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:23.516544    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:14:25.067728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:25.067681    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"]
Nov 03 19:14:25.106908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:25.106871    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-904c41b6-n857f"]
Nov 03 19:14:25.936634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:25.936600    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee201c8-5c45-46ba-b56d-b996d45fcab4" path="/var/lib/kubelet/pods/5ee201c8-5c45-46ba-b56d-b996d45fcab4/volumes"
Nov 03 19:14:26.873067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:26.873028    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:27.118241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:27.118197    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:28.047214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:28.047182    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"]
Nov 03 19:14:28.093497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:28.093464    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83776904-fljhh"]
Nov 03 19:14:29.936228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:29.936194    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="985dc89e-f210-45f9-8ab9-bcee80d6cb6b" path="/var/lib/kubelet/pods/985dc89e-f210-45f9-8ab9-bcee80d6cb6b/volumes"
Nov 03 19:14:31.486312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.486277    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"]
Nov 03 19:14:31.487116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.487099    4944 cpu_manager.go:414] "containerMap: removing container" podUID="20393455-57b1-4c20-ac21-682b66815215" containerName="revoke-keys"
Nov 03 19:14:31.487174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.487117    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20393455-57b1-4c20-ac21-682b66815215" containerName="revoke-keys"
Nov 03 19:14:31.487231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.487222    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="20393455-57b1-4c20-ac21-682b66815215" containerName="revoke-keys"
Nov 03 19:14:31.491152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.491135    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.540203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.540165    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:14:31.544577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.544549    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"]
Nov 03 19:14:31.585091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.585062    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:14:31.585239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.585112    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:14:31.585239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.585196    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:14:31.628060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.628023    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x8gf\" (UniqueName: \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.628241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.628174    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.628329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.628308    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.748589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.748465    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x8gf\" (UniqueName: \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.751989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.751954    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.752189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.752137    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.753264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.753233    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.757941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.757907    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:31.788883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.788851    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:31.808976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:31.808938    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x8gf\" (UniqueName: \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\") pod \"maas-api-revoke-sub-0c6e520a-xcjqx\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:32.049844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:32.049725    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"]
Nov 03 19:14:32.053853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:32.053827    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b780ca8-tb7qp"]
Nov 03 19:14:32.105559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:32.105526    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:32.490395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:32.490364    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"]
Nov 03 19:14:32.494360 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:14:32.494331    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e0a356_3c66_49df_8b8c_8d5729701257.slice/crio-7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6 WatchSource:0}: Error finding container 7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6: Status 404 returned error can't find the container with id 7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6
Nov 03 19:14:32.718010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:32.717973    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx" event={"ID":"80e0a356-3c66-49df-8b8c-8d5729701257","Type":"ContainerStarted","Data":"7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6"}
Nov 03 19:14:33.728461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:33.728427    4944 generic.go:356] "Generic (PLEG): container finished" podID="80e0a356-3c66-49df-8b8c-8d5729701257" containerID="d195594067ab1ca8f06d140a2fb1ea0235c2fafe31fcc0b721b9d60889803f68" exitCode=0
Nov 03 19:14:33.728887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:33.728516    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx" event={"ID":"80e0a356-3c66-49df-8b8c-8d5729701257","Type":"ContainerDied","Data":"d195594067ab1ca8f06d140a2fb1ea0235c2fafe31fcc0b721b9d60889803f68"}
Nov 03 19:14:33.936873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:33.936801    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce71709-d05b-4ac1-8864-a1bf0b352042" path="/var/lib/kubelet/pods/1ce71709-d05b-4ac1-8864-a1bf0b352042/volumes"
Nov 03 19:14:34.936105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:34.936079    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:35.015133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.015100    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\") pod \"80e0a356-3c66-49df-8b8c-8d5729701257\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") "
Nov 03 19:14:35.015309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.015289    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\") pod \"80e0a356-3c66-49df-8b8c-8d5729701257\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") "
Nov 03 19:14:35.015408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.015389    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\" (UniqueName: \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\") pod \"80e0a356-3c66-49df-8b8c-8d5729701257\" (UID: \"80e0a356-3c66-49df-8b8c-8d5729701257\") "
Nov 03 19:14:35.015508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.015489    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca" pod "80e0a356-3c66-49df-8b8c-8d5729701257" (UID: "80e0a356-3c66-49df-8b8c-8d5729701257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:14:35.016432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.016402    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:35.017555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.017531    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca" pod "80e0a356-3c66-49df-8b8c-8d5729701257" (UID: "80e0a356-3c66-49df-8b8c-8d5729701257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:14:35.017652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.017603    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf" pod "80e0a356-3c66-49df-8b8c-8d5729701257" (UID: "80e0a356-3c66-49df-8b8c-8d5729701257"). InnerVolumeSpecName "kube-api-access-2x8gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:14:35.117985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.117946    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80e0a356-3c66-49df-8b8c-8d5729701257-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:35.117985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.117978    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x8gf\" (UniqueName: \"kubernetes.io/projected/80e0a356-3c66-49df-8b8c-8d5729701257-kube-api-access-2x8gf\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:35.226841 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:14:35.226770    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:14:35.756629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.756123    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx" event={"ID":"80e0a356-3c66-49df-8b8c-8d5729701257","Type":"ContainerDied","Data":"7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6"}
Nov 03 19:14:35.756629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.756174    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eccbd0c875d2172d9b4c364fbd092ca94ade22a6a96d2c5d181741bc3de72b6"
Nov 03 19:14:35.756629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:35.756226    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"
Nov 03 19:14:36.058678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:36.058593    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"]
Nov 03 19:14:36.062680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:36.062650    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b29e321-mjtd4"]
Nov 03 19:14:36.923584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:36.923551    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:37.171620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:37.171583    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:37.670611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:37.670576    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:37.938496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:37.938397    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15764425-3362-4021-966e-a939053ad959" path="/var/lib/kubelet/pods/15764425-3362-4021-966e-a939053ad959/volumes"
Nov 03 19:14:40.637122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:40.637080    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:42.207557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:42.207517    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:42.871103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:42.871067    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:43.962245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:43.962211    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:46.603840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:46.603806    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:14:46.604262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:46.604021    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy" containerID="cri-o://c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614" gracePeriod=30
Nov 03 19:14:47.563591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:47.563552    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:48.508997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:48.508967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:51.420350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.420316    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:51.678997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.678915    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:51.887454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.887426    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:14:51.916823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.916746    4944 generic.go:356] "Generic (PLEG): container finished" podID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerID="c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614" exitCode=0
Nov 03 19:14:51.917089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.916843    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"
Nov 03 19:14:51.917089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.916839    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" event={"ID":"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b","Type":"ContainerDied","Data":"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"}
Nov 03 19:14:51.917089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.916908    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm" event={"ID":"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b","Type":"ContainerDied","Data":"3a603bdbd01e39e206d7138e72ab7b51307898e1170e4e485d1b34f2911b782a"}
Nov 03 19:14:51.917089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.916932    4944 scope.go:122] "RemoveContainer" containerID="c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"
Nov 03 19:14:51.932122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.932097    4944 scope.go:122] "RemoveContainer" containerID="c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"
Nov 03 19:14:51.932434 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:14:51.932413    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614\": container with ID starting with c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614 not found: ID does not exist" containerID="c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"
Nov 03 19:14:51.932489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:51.932445    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614"} err="failed to get container status \"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614\": rpc error: code = NotFound desc = could not find container \"c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614\": container with ID starting with c7eb67b1a7c8869c23bdbac70d99d906c756cd735bcfa4f75f3a754b592d5614 not found: ID does not exist"
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.015539    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.015607    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.015829    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.015897    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.016027    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.016070    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.016110    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.016187    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.018818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.016313    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\") pod \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\" (UID: \"9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b\") "
Nov 03 19:14:52.039021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.033400    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:14:52.039021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.036085    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:14:52.039021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.036345    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:14:52.039021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.036507    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:14:52.069079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.062935    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:14:52.069079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.062946    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:14:52.069079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.063025    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:14:52.104683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.104004    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "kube-api-access-8wd6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:14:52.104683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.104304    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token" pod "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" (UID: "9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124541    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124579    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124594    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wd6r\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-kube-api-access-8wd6r\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124608    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124620    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124633    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124646    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124660    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.131198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.124677    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:14:52.237053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.236911    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"]
Nov 03 19:14:52.249487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.249446    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a149041e-zdrpr"]
Nov 03 19:14:52.357237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.357203    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:14:52.361523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.361495    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-7nznm"]
Nov 03 19:14:52.436814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:52.436768    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:14:53.941878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:53.941839    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412258e2-cc67-43f7-998f-28c95092e9cf" path="/var/lib/kubelet/pods/412258e2-cc67-43f7-998f-28c95092e9cf/volumes"
Nov 03 19:14:53.942672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:53.942648    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" path="/var/lib/kubelet/pods/9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b/volumes"
Nov 03 19:14:57.500398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:14:57.500364    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:00.171835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.171763    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:00.172979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.172958    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy"
Nov 03 19:15:00.173052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.172983    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy"
Nov 03 19:15:00.173092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.173063    4944 cpu_manager.go:414] "containerMap: removing container" podUID="80e0a356-3c66-49df-8b8c-8d5729701257" containerName="revoke-keys"
Nov 03 19:15:00.173092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.173072    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e0a356-3c66-49df-8b8c-8d5729701257" containerName="revoke-keys"
Nov 03 19:15:00.173266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.173253    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e0a356-3c66-49df-8b8c-8d5729701257" containerName="revoke-keys"
Nov 03 19:15:00.173307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.173277    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d1f81f7-c4f0-469f-87e8-ccda6a51ee3b" containerName="istio-proxy"
Nov 03 19:15:00.179156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.179133    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:00.179261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.179244    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.184063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.184039    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:00.184195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.184137    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.188516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.188498    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.218237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.218215    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:00.222627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.222608    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:00.222754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.222734    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.226926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.226902    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:00.227031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.227011    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.237869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.237847    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:00.239928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.239907    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.245446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.245426    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:00.245564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.245532    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.253189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.253170    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:00.253284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.253271    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.258397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.258377    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.260509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.260488    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:15:00.260509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.260503    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:15:00.260926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.260906    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:15:00.263586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.263561    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8cf9\" (UniqueName: \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.263811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.263780    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.266014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.265988    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:00.271237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.271213    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:00.312195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.312166    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:00.317741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.317711    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:00.323493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.323170    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:00.331100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.331075    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:00.333680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.333649    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:00.362031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.362002    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.366367    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr42c\" (UniqueName: \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.366555    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmdks\" (UniqueName: \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.366703    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.366766    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.367564    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.367640    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.367825    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rh8n\" (UniqueName: \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.367858    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.367959    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368004    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368103    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gswtb\" (UniqueName: \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368129    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368166    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368199    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368239    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8cf9\" (UniqueName: \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368315    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97xgf\" (UniqueName: \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368357    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmn4q\" (UniqueName: \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368382    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx94l\" (UniqueName: \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.374880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.368450    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9bbq\" (UniqueName: \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.383228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.382568    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:00.425781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.425719    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8cf9\" (UniqueName: \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.469424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469389    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr42c\" (UniqueName: \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.469581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469435    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmdks\" (UniqueName: \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.469581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469485    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.469682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469606    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.469761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469742    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rh8n\" (UniqueName: \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.469998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469781    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.469998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469853    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.469998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469883    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.469998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.469986    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gswtb\" (UniqueName: \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.470243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470016    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.470243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470051    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.470243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470083    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.470243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470177    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97xgf\" (UniqueName: \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.470243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470219    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmn4q\" (UniqueName: \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.470517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470244    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx94l\" (UniqueName: \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.470517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470318    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9bbq\" (UniqueName: \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.470627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470521    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.470686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470656    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.470744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470685    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.470744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.470728    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.471038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.471010    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.471224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.471207    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.471224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.471217    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.471347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.471268    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.495335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.495288    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:00.527683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.527655    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmdks\" (UniqueName: \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.528221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.528156    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr42c\" (UniqueName: \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\") pod \"maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.528917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.528343    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rh8n\" (UniqueName: \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\") pod \"maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.528917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.528475    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9bbq\" (UniqueName: \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\") pod \"maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.528917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.528877    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gswtb\" (UniqueName: \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.542046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.542013    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmn4q\" (UniqueName: \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\") pod \"maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.542448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.542423    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx94l\" (UniqueName: \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\") pod \"maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.543215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.543169    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97xgf\" (UniqueName: \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\") pod \"maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.548017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.547996    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:00.558252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.558073    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:00.568454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.568225    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:00.580515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.580189    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:00.588875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.588530    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:00.596159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.596138    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:00.807454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.807398    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:00.818381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.812504    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:00.889771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:00.889733    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:00.897983 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:00.897897    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd2bba4_2d87_4520_9b59_d9236e88c3df.slice/crio-0236da1c731a6a1e8447427e25292dcca0f8795a28c7ec181173d09ecb62dfe3 WatchSource:0}: Error finding container 0236da1c731a6a1e8447427e25292dcca0f8795a28c7ec181173d09ecb62dfe3: Status 404 returned error can't find the container with id 0236da1c731a6a1e8447427e25292dcca0f8795a28c7ec181173d09ecb62dfe3
Nov 03 19:15:01.020783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.020739    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerStarted","Data":"0236da1c731a6a1e8447427e25292dcca0f8795a28c7ec181173d09ecb62dfe3"}
Nov 03 19:15:01.197094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.197058    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:01.213083 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.213029    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b1c257_9976_400d_b881_b6f71c959b0e.slice/crio-7eef10363e1893b17285f50cae1be042a38bb089b46632b1238eb5ecfad45229 WatchSource:0}: Error finding container 7eef10363e1893b17285f50cae1be042a38bb089b46632b1238eb5ecfad45229: Status 404 returned error can't find the container with id 7eef10363e1893b17285f50cae1be042a38bb089b46632b1238eb5ecfad45229
Nov 03 19:15:01.214673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.214646    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:01.317963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.317912    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:01.322364 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.322331    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f9b167_ec6c_449f_b240_fa3072a1e2a9.slice/crio-cdab178e80097e0a7b20b068d5fa097c18cc091e872716eb3047ded0371a9457 WatchSource:0}: Error finding container cdab178e80097e0a7b20b068d5fa097c18cc091e872716eb3047ded0371a9457: Status 404 returned error can't find the container with id cdab178e80097e0a7b20b068d5fa097c18cc091e872716eb3047ded0371a9457
Nov 03 19:15:01.323178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.323151    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:01.326842 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.326783    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f10652_ddb3_4d46_b5f7_c9e4c26ff448.slice/crio-74120c19c5b78e420bc5c164d8aa4bd16637611363a3e298076ea082528a2234 WatchSource:0}: Error finding container 74120c19c5b78e420bc5c164d8aa4bd16637611363a3e298076ea082528a2234: Status 404 returned error can't find the container with id 74120c19c5b78e420bc5c164d8aa4bd16637611363a3e298076ea082528a2234
Nov 03 19:15:01.328570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.328544    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:01.329030 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.329012    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543790ca_2db0_4851_8cc0_20ebb3d99160.slice/crio-868a819ca107aad5171ea32d7091c6d5c86f28106af9588b90c5a34f0efbee1f WatchSource:0}: Error finding container 868a819ca107aad5171ea32d7091c6d5c86f28106af9588b90c5a34f0efbee1f: Status 404 returned error can't find the container with id 868a819ca107aad5171ea32d7091c6d5c86f28106af9588b90c5a34f0efbee1f
Nov 03 19:15:01.418717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.418691    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:01.426561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.426540    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:01.430918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.430894    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:01.514421 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.514344    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74683d6c_0f1d_45ae_83e6_e73b726a3fe1.slice/crio-5eaf51967f4d3568e72636e95d23202a9dd6a7c0bda7792a14e81fd90c08ca76 WatchSource:0}: Error finding container 5eaf51967f4d3568e72636e95d23202a9dd6a7c0bda7792a14e81fd90c08ca76: Status 404 returned error can't find the container with id 5eaf51967f4d3568e72636e95d23202a9dd6a7c0bda7792a14e81fd90c08ca76
Nov 03 19:15:01.515259 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:01.515217    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d692a4_9aeb_463a_91a3_fcc56157ea48.slice/crio-e5334b6801c7483f77a4b1d5fb2ec9e6bf5663601c6e2e89350724cb57f2f58c WatchSource:0}: Error finding container e5334b6801c7483f77a4b1d5fb2ec9e6bf5663601c6e2e89350724cb57f2f58c: Status 404 returned error can't find the container with id e5334b6801c7483f77a4b1d5fb2ec9e6bf5663601c6e2e89350724cb57f2f58c
Nov 03 19:15:01.660486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.660452    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:15:01.660766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:01.660717    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy" containerID="cri-o://66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07" gracePeriod=30
Nov 03 19:15:02.053393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.052046    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerStarted","Data":"63b5b9deafdb9b5d9f52be13c2ebb2b57d84bf43c76b38ff2e1e90b895e814a5"}
Nov 03 19:15:02.053393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.052125    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerStarted","Data":"cdab178e80097e0a7b20b068d5fa097c18cc091e872716eb3047ded0371a9457"}
Nov 03 19:15:02.059284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.059150    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerStarted","Data":"5eaf51967f4d3568e72636e95d23202a9dd6a7c0bda7792a14e81fd90c08ca76"}
Nov 03 19:15:02.062681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.062648    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerStarted","Data":"9f64ab53ef8724f48efc9cb875b415d041a4c00ced6a8b20f03012251fe26fba"}
Nov 03 19:15:02.065217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.065181    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:02.068413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.068348    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerStarted","Data":"868a819ca107aad5171ea32d7091c6d5c86f28106af9588b90c5a34f0efbee1f"}
Nov 03 19:15:02.072574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.072544    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerStarted","Data":"74120c19c5b78e420bc5c164d8aa4bd16637611363a3e298076ea082528a2234"}
Nov 03 19:15:02.075953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.075917    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerStarted","Data":"0c28fe692b50f8d0fbfdbfceb09b1e0373f5f43ec2c2f86be4d30adef1c3e483"}
Nov 03 19:15:02.076467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.076436    4944 scope.go:122] "RemoveContainer" containerID="0c28fe692b50f8d0fbfdbfceb09b1e0373f5f43ec2c2f86be4d30adef1c3e483"
Nov 03 19:15:02.078869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.078842    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" event={"ID":"d6b1c257-9976-400d-b881-b6f71c959b0e","Type":"ContainerStarted","Data":"7eef10363e1893b17285f50cae1be042a38bb089b46632b1238eb5ecfad45229"}
Nov 03 19:15:02.080667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.080640    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" event={"ID":"36d692a4-9aeb-463a-91a3-fcc56157ea48","Type":"ContainerStarted","Data":"e5334b6801c7483f77a4b1d5fb2ec9e6bf5663601c6e2e89350724cb57f2f58c"}
Nov 03 19:15:02.088405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.088365    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerStarted","Data":"9370184f4bd747f523ddf684b3923aabbdcbc07bb4cc6e51b9f216bdd194b4d7"}
Nov 03 19:15:02.088540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.088484    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerStarted","Data":"98381b99a536db3ed4ef083e2ac022b773ec14da36b0aa794d01881127f025fc"}
Nov 03 19:15:02.164217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.164157    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" podStartSLOduration=2.164138994 podStartE2EDuration="2.164138994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.160571963 +0000 UTC m=+1800.437925550" watchObservedRunningTime="2026-11-03 19:15:02.164138994 +0000 UTC m=+1800.441492587"
Nov 03 19:15:02.193391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.193362    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:15:02.196744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.196718    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:15:02.273819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:02.273756    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" podStartSLOduration=2.273742915 podStartE2EDuration="2.273742915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:02.268549671 +0000 UTC m=+1800.545903267" watchObservedRunningTime="2026-11-03 19:15:02.273742915 +0000 UTC m=+1800.551096498"
Nov 03 19:15:03.098637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.098598    4944 scope.go:122] "RemoveContainer" containerID="e44c1bee26b78c9115402e4429f83bd089627cb73c3149a9c5dbbccdf99263c0"
Nov 03 19:15:03.108457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.108421    4944 generic.go:356] "Generic (PLEG): container finished" podID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerID="63b5b9deafdb9b5d9f52be13c2ebb2b57d84bf43c76b38ff2e1e90b895e814a5" exitCode=60
Nov 03 19:15:03.108592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.108515    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerDied","Data":"63b5b9deafdb9b5d9f52be13c2ebb2b57d84bf43c76b38ff2e1e90b895e814a5"}
Nov 03 19:15:03.108863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.108845    4944 scope.go:122] "RemoveContainer" containerID="63b5b9deafdb9b5d9f52be13c2ebb2b57d84bf43c76b38ff2e1e90b895e814a5"
Nov 03 19:15:03.110423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.110400    4944 generic.go:356] "Generic (PLEG): container finished" podID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerID="60d3ce8d6b8e35847d6aaca11ed1b0a8d8f0df5289b61612a6c63d76faccb913" exitCode=60
Nov 03 19:15:03.110542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.110484    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerDied","Data":"60d3ce8d6b8e35847d6aaca11ed1b0a8d8f0df5289b61612a6c63d76faccb913"}
Nov 03 19:15:03.110764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.110747    4944 scope.go:122] "RemoveContainer" containerID="60d3ce8d6b8e35847d6aaca11ed1b0a8d8f0df5289b61612a6c63d76faccb913"
Nov 03 19:15:03.114188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.114168    4944 scope.go:122] "RemoveContainer" containerID="017a179ac1d75fbdffa9004c37adfe045b18c3a37573af3c3ba19d63c332e819"
Nov 03 19:15:03.117099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.117079    4944 generic.go:356] "Generic (PLEG): container finished" podID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerID="2293f8506ecd4188fb3a5f64fd2cf1606f92fd3bf270f053093f2301c0f6af14" exitCode=60
Nov 03 19:15:03.117175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.117163    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerDied","Data":"2293f8506ecd4188fb3a5f64fd2cf1606f92fd3bf270f053093f2301c0f6af14"}
Nov 03 19:15:03.117485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.117468    4944 scope.go:122] "RemoveContainer" containerID="2293f8506ecd4188fb3a5f64fd2cf1606f92fd3bf270f053093f2301c0f6af14"
Nov 03 19:15:03.119268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.119249    4944 generic.go:356] "Generic (PLEG): container finished" podID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerID="508d7162bf700722ba767636d8eefca8263d37883a2288e2dfc7e45a3c378770" exitCode=60
Nov 03 19:15:03.119366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.119338    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerDied","Data":"508d7162bf700722ba767636d8eefca8263d37883a2288e2dfc7e45a3c378770"}
Nov 03 19:15:03.119533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.119519    4944 scope.go:122] "RemoveContainer" containerID="508d7162bf700722ba767636d8eefca8263d37883a2288e2dfc7e45a3c378770"
Nov 03 19:15:03.120943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.120925    4944 generic.go:356] "Generic (PLEG): container finished" podID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerID="8780d9f09d46dc784664c7a206576b836bf0b882bb837daaae9281e62e1d9a9c" exitCode=60
Nov 03 19:15:03.121031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.120986    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerDied","Data":"8780d9f09d46dc784664c7a206576b836bf0b882bb837daaae9281e62e1d9a9c"}
Nov 03 19:15:03.121305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.121285    4944 scope.go:122] "RemoveContainer" containerID="8780d9f09d46dc784664c7a206576b836bf0b882bb837daaae9281e62e1d9a9c"
Nov 03 19:15:03.123067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.123049    4944 generic.go:356] "Generic (PLEG): container finished" podID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerID="0c28fe692b50f8d0fbfdbfceb09b1e0373f5f43ec2c2f86be4d30adef1c3e483" exitCode=60
Nov 03 19:15:03.123167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.123127    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerDied","Data":"0c28fe692b50f8d0fbfdbfceb09b1e0373f5f43ec2c2f86be4d30adef1c3e483"}
Nov 03 19:15:03.123225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.123170    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerStarted","Data":"fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"}
Nov 03 19:15:03.125227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.125062    4944 generic.go:356] "Generic (PLEG): container finished" podID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerID="1e0f88cdbdb1559aec13aa1c9d5765826049f518db48646b1c81c2808358c139" exitCode=60
Nov 03 19:15:03.125227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.125185    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" event={"ID":"d6b1c257-9976-400d-b881-b6f71c959b0e","Type":"ContainerDied","Data":"1e0f88cdbdb1559aec13aa1c9d5765826049f518db48646b1c81c2808358c139"}
Nov 03 19:15:03.125443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.125425    4944 scope.go:122] "RemoveContainer" containerID="1e0f88cdbdb1559aec13aa1c9d5765826049f518db48646b1c81c2808358c139"
Nov 03 19:15:03.128321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.128303    4944 generic.go:356] "Generic (PLEG): container finished" podID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerID="a9d885b4ca2c4e478a2d77a7792c7db6f7c0d94479bb840e0a4076bd1b156931" exitCode=60
Nov 03 19:15:03.128382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.128366    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" event={"ID":"36d692a4-9aeb-463a-91a3-fcc56157ea48","Type":"ContainerDied","Data":"a9d885b4ca2c4e478a2d77a7792c7db6f7c0d94479bb840e0a4076bd1b156931"}
Nov 03 19:15:03.128675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.128659    4944 scope.go:122] "RemoveContainer" containerID="a9d885b4ca2c4e478a2d77a7792c7db6f7c0d94479bb840e0a4076bd1b156931"
Nov 03 19:15:03.130502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.130481    4944 generic.go:356] "Generic (PLEG): container finished" podID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerID="9370184f4bd747f523ddf684b3923aabbdcbc07bb4cc6e51b9f216bdd194b4d7" exitCode=60
Nov 03 19:15:03.130594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.130560    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerDied","Data":"9370184f4bd747f523ddf684b3923aabbdcbc07bb4cc6e51b9f216bdd194b4d7"}
Nov 03 19:15:03.130763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.130743    4944 scope.go:122] "RemoveContainer" containerID="9370184f4bd747f523ddf684b3923aabbdcbc07bb4cc6e51b9f216bdd194b4d7"
Nov 03 19:15:03.140551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.140536    4944 scope.go:122] "RemoveContainer" containerID="d7403fc90ab63e95604053d504554c1d2c82a4038fb6ea90c888e4cd220a2d33"
Nov 03 19:15:03.159439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.159419    4944 scope.go:122] "RemoveContainer" containerID="cff864f33a868f3b24c32399803c0721e6cf27a4f022187f1775d3c964b1ebb9"
Nov 03 19:15:03.187959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.187936    4944 scope.go:122] "RemoveContainer" containerID="48ce52597929a13d893226ed39cce7f28ae450fb5b55d8639eface471879ce48"
Nov 03 19:15:03.317011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.316993    4944 scope.go:122] "RemoveContainer" containerID="171189afe9f66a17467e5801485fac0b028fa15baec30c3d2fe53572c157ca60"
Nov 03 19:15:03.423399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.423270    4944 scope.go:122] "RemoveContainer" containerID="fdbd9d49fd777ef4a8d3e90a66a99c57c6293f86cb95415bf12a1a7134ef4880"
Nov 03 19:15:03.434592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.434573    4944 scope.go:122] "RemoveContainer" containerID="4af5ff372fbaea42c226f55b734dd77725b93006a8945f4b77e35259128aaba8"
Nov 03 19:15:03.448312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.448282    4944 scope.go:122] "RemoveContainer" containerID="0528e8b4f3706c1af950723cb259dfa18c304406113472eaf7f9a41c26daedd0"
Nov 03 19:15:03.457839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.457819    4944 scope.go:122] "RemoveContainer" containerID="71bdfe0078163e9fbdb6c310e23725cf60960d7297b058c8e8ac862cca541c15"
Nov 03 19:15:03.467765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.467746    4944 scope.go:122] "RemoveContainer" containerID="795cf2ceec25d2a2fb29290f0b537f708c5b90f90d177bdcc71ec50cf6cb2766"
Nov 03 19:15:03.477164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.477143    4944 scope.go:122] "RemoveContainer" containerID="fe4b3849efb911f8553526a7f8d264a8c00aac73b2f8de0ae982b80488a0fa1b"
Nov 03 19:15:03.486363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.486332    4944 scope.go:122] "RemoveContainer" containerID="3e0c1d7d8195ae2b0b66694c593bf36754d75387a3be71acddd61fd2c30e0430"
Nov 03 19:15:03.653803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.652970    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" podStartSLOduration=3.652955764 podStartE2EDuration="3.652955764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:03.648424468 +0000 UTC m=+1801.925778052" watchObservedRunningTime="2026-11-03 19:15:03.652955764 +0000 UTC m=+1801.930309371"
Nov 03 19:15:03.748466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.748441    4944 scope.go:122] "RemoveContainer" containerID="27e486c63fde249103db4e54e6e496b109dd37a66766294bda882e4e6d7438c4"
Nov 03 19:15:03.776861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:03.776839    4944 scope.go:122] "RemoveContainer" containerID="26d8408480b3a80caed989da0bb577fe4117038d78e675355a8bcc9913ddc92a"
Nov 03 19:15:04.146405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.146368    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerStarted","Data":"4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"}
Nov 03 19:15:04.148365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.148328    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerStarted","Data":"4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"}
Nov 03 19:15:04.150173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.150152    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerStarted","Data":"83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"}
Nov 03 19:15:04.151917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.151895    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerStarted","Data":"b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"}
Nov 03 19:15:04.153635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.153615    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerStarted","Data":"e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"}
Nov 03 19:15:04.155439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.155419    4944 generic.go:356] "Generic (PLEG): container finished" podID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerID="fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7" exitCode=60
Nov 03 19:15:04.155521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.155499    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerDied","Data":"fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"}
Nov 03 19:15:04.155607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.155594    4944 scope.go:122] "RemoveContainer" containerID="0c28fe692b50f8d0fbfdbfceb09b1e0373f5f43ec2c2f86be4d30adef1c3e483"
Nov 03 19:15:04.155713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.155696    4944 scope.go:122] "RemoveContainer" containerID="fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"
Nov 03 19:15:04.155999 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:04.155981    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c_odh-ai-gateway-infra(ddd2bba4-2d87-4520-9b59-d9236e88c3df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df"
Nov 03 19:15:04.363960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.363912    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" podStartSLOduration=4.363896054 podStartE2EDuration="4.363896054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.363158742 +0000 UTC m=+1802.640512338" watchObservedRunningTime="2026-11-03 19:15:04.363896054 +0000 UTC m=+1802.641249637"
Nov 03 19:15:04.470832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.470761    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" podStartSLOduration=4.47074369 podStartE2EDuration="4.47074369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.470678007 +0000 UTC m=+1802.748031591" watchObservedRunningTime="2026-11-03 19:15:04.47074369 +0000 UTC m=+1802.748097274"
Nov 03 19:15:04.687193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.687147    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" podStartSLOduration=4.687134582 podStartE2EDuration="4.687134582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.685785079 +0000 UTC m=+1802.963138661" watchObservedRunningTime="2026-11-03 19:15:04.687134582 +0000 UTC m=+1802.964488164"
Nov 03 19:15:04.796137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:04.796085    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" podStartSLOduration=4.796069826 podStartE2EDuration="4.796069826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:04.792526218 +0000 UTC m=+1803.069879803" watchObservedRunningTime="2026-11-03 19:15:04.796069826 +0000 UTC m=+1803.073423437"
Nov 03 19:15:05.166626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.166549    4944 generic.go:356] "Generic (PLEG): container finished" podID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerID="4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e" exitCode=60
Nov 03 19:15:05.166814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.166629    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerDied","Data":"4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"}
Nov 03 19:15:05.166814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.166728    4944 scope.go:122] "RemoveContainer" containerID="63b5b9deafdb9b5d9f52be13c2ebb2b57d84bf43c76b38ff2e1e90b895e814a5"
Nov 03 19:15:05.166963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.166943    4944 scope.go:122] "RemoveContainer" containerID="4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"
Nov 03 19:15:05.167315 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.167292    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5_odh-ai-gateway-infra(10f9b167-ec6c-449f-b240-fa3072a1e2a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9"
Nov 03 19:15:05.169241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.169223    4944 generic.go:356] "Generic (PLEG): container finished" podID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerID="4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc" exitCode=60
Nov 03 19:15:05.169334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.169285    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerDied","Data":"4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"}
Nov 03 19:15:05.169605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.169583    4944 scope.go:122] "RemoveContainer" containerID="4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"
Nov 03 19:15:05.170018 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.169951    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2_odh-ai-gateway-infra(74683d6c-0f1d-45ae-83e6-e73b726a3fe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1"
Nov 03 19:15:05.171173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.171155    4944 generic.go:356] "Generic (PLEG): container finished" podID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerID="83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927" exitCode=60
Nov 03 19:15:05.171254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.171219    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerDied","Data":"83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"}
Nov 03 19:15:05.171466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.171449    4944 scope.go:122] "RemoveContainer" containerID="83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"
Nov 03 19:15:05.171733 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.171711    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc_odh-ai-gateway-infra(773df7ae-91e4-4b53-94fd-d6f257e50cde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde"
Nov 03 19:15:05.173107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.173088    4944 generic.go:356] "Generic (PLEG): container finished" podID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerID="b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd" exitCode=60
Nov 03 19:15:05.173208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.173171    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerDied","Data":"b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"}
Nov 03 19:15:05.173392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.173375    4944 scope.go:122] "RemoveContainer" containerID="b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"
Nov 03 19:15:05.173639 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.173621    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l_odh-ai-gateway-infra(543790ca-2db0-4851-8cc0-20ebb3d99160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160"
Nov 03 19:15:05.175158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.175141    4944 generic.go:356] "Generic (PLEG): container finished" podID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerID="e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059" exitCode=60
Nov 03 19:15:05.175251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.175197    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerDied","Data":"e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"}
Nov 03 19:15:05.175436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.175420    4944 scope.go:122] "RemoveContainer" containerID="e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"
Nov 03 19:15:05.175697 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.175676    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w_odh-ai-gateway-infra(e7f10652-ddb3-4d46-b5f7-c9e4c26ff448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448"
Nov 03 19:15:05.177110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.177092    4944 scope.go:122] "RemoveContainer" containerID="fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"
Nov 03 19:15:05.177339 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.177317    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c_odh-ai-gateway-infra(ddd2bba4-2d87-4520-9b59-d9236e88c3df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df"
Nov 03 19:15:05.178650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.178633    4944 generic.go:356] "Generic (PLEG): container finished" podID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerID="d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab" exitCode=60
Nov 03 19:15:05.178741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.178713    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" event={"ID":"d6b1c257-9976-400d-b881-b6f71c959b0e","Type":"ContainerDied","Data":"d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab"}
Nov 03 19:15:05.178944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.178929    4944 scope.go:122] "RemoveContainer" containerID="d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab"
Nov 03 19:15:05.179176 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.179155    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw_odh-ai-gateway-infra(d6b1c257-9976-400d-b881-b6f71c959b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e"
Nov 03 19:15:05.180838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.180819    4944 generic.go:356] "Generic (PLEG): container finished" podID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerID="eead4dbcf9b5fe4c30c7fdebb5635c05356f5a2aa45028f37889b7c1159b3543" exitCode=60
Nov 03 19:15:05.180907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.180884    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" event={"ID":"36d692a4-9aeb-463a-91a3-fcc56157ea48","Type":"ContainerDied","Data":"eead4dbcf9b5fe4c30c7fdebb5635c05356f5a2aa45028f37889b7c1159b3543"}
Nov 03 19:15:05.181084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.181070    4944 scope.go:122] "RemoveContainer" containerID="eead4dbcf9b5fe4c30c7fdebb5635c05356f5a2aa45028f37889b7c1159b3543"
Nov 03 19:15:05.181340 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.181310    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg_odh-ai-gateway-infra(36d692a4-9aeb-463a-91a3-fcc56157ea48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48"
Nov 03 19:15:05.182892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.182804    4944 generic.go:356] "Generic (PLEG): container finished" podID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerID="8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d" exitCode=60
Nov 03 19:15:05.182955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.182888    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerDied","Data":"8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d"}
Nov 03 19:15:05.182955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.182927    4944 scope.go:122] "RemoveContainer" containerID="60d3ce8d6b8e35847d6aaca11ed1b0a8d8f0df5289b61612a6c63d76faccb913"
Nov 03 19:15:05.183095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.183081    4944 scope.go:122] "RemoveContainer" containerID="8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d"
Nov 03 19:15:05.183312 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.183294    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz_odh-ai-gateway-infra(3b1cdd76-fe72-43be-aa6f-b9674622c979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979"
Nov 03 19:15:05.196555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.196540    4944 scope.go:122] "RemoveContainer" containerID="2293f8506ecd4188fb3a5f64fd2cf1606f92fd3bf270f053093f2301c0f6af14"
Nov 03 19:15:05.211953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.211932    4944 scope.go:122] "RemoveContainer" containerID="508d7162bf700722ba767636d8eefca8263d37883a2288e2dfc7e45a3c378770"
Nov 03 19:15:05.226676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.226655    4944 scope.go:122] "RemoveContainer" containerID="8780d9f09d46dc784664c7a206576b836bf0b882bb837daaae9281e62e1d9a9c"
Nov 03 19:15:05.228049 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:05.228026    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:15:05.239967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.239951    4944 scope.go:122] "RemoveContainer" containerID="1e0f88cdbdb1559aec13aa1c9d5765826049f518db48646b1c81c2808358c139"
Nov 03 19:15:05.253246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.253227    4944 scope.go:122] "RemoveContainer" containerID="a9d885b4ca2c4e478a2d77a7792c7db6f7c0d94479bb840e0a4076bd1b156931"
Nov 03 19:15:05.266761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:05.266740    4944 scope.go:122] "RemoveContainer" containerID="9370184f4bd747f523ddf684b3923aabbdcbc07bb4cc6e51b9f216bdd194b4d7"
Nov 03 19:15:06.192350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.192319    4944 scope.go:122] "RemoveContainer" containerID="83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"
Nov 03 19:15:06.192740 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.192659    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc_odh-ai-gateway-infra(773df7ae-91e4-4b53-94fd-d6f257e50cde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde"
Nov 03 19:15:06.193987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.193966    4944 scope.go:122] "RemoveContainer" containerID="b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"
Nov 03 19:15:06.194302 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.194278    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l_odh-ai-gateway-infra(543790ca-2db0-4851-8cc0-20ebb3d99160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160"
Nov 03 19:15:06.195485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.195461    4944 scope.go:122] "RemoveContainer" containerID="e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"
Nov 03 19:15:06.195761 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.195739    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w_odh-ai-gateway-infra(e7f10652-ddb3-4d46-b5f7-c9e4c26ff448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448"
Nov 03 19:15:06.197083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.197063    4944 scope.go:122] "RemoveContainer" containerID="d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab"
Nov 03 19:15:06.197381 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.197352    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw_odh-ai-gateway-infra(d6b1c257-9976-400d-b881-b6f71c959b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e"
Nov 03 19:15:06.198634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.198608    4944 scope.go:122] "RemoveContainer" containerID="eead4dbcf9b5fe4c30c7fdebb5635c05356f5a2aa45028f37889b7c1159b3543"
Nov 03 19:15:06.198900 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.198880    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg_odh-ai-gateway-infra(36d692a4-9aeb-463a-91a3-fcc56157ea48)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48"
Nov 03 19:15:06.200054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.200037    4944 scope.go:122] "RemoveContainer" containerID="8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d"
Nov 03 19:15:06.200303 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.200276    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz_odh-ai-gateway-infra(3b1cdd76-fe72-43be-aa6f-b9674622c979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979"
Nov 03 19:15:06.201645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.201626    4944 scope.go:122] "RemoveContainer" containerID="4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"
Nov 03 19:15:06.201910 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.201889    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5_odh-ai-gateway-infra(10f9b167-ec6c-449f-b240-fa3072a1e2a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9"
Nov 03 19:15:06.203056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.203037    4944 scope.go:122] "RemoveContainer" containerID="4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"
Nov 03 19:15:06.203258 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:06.203241    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2_odh-ai-gateway-infra(74683d6c-0f1d-45ae-83e6-e73b726a3fe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1"
Nov 03 19:15:06.727304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.727274    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:06.952214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.952143    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:15:06.993930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.993897    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.993953    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994055    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994093    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994140    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994181    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:06.994299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994238    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994309    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994399    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994485    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\") pod \"2226d542-31a5-4042-8aa7-43b49d0e5bac\" (UID: \"2226d542-31a5-4042-8aa7-43b49d0e5bac\") "
Nov 03 19:15:06.994564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994499    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:06.994731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994710    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:06.994847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994768    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:06.994847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.994784    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:06.996963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.996679    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "kube-api-access-rbckt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:06.997125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997103    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997130    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997147    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbckt\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-kube-api-access-rbckt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997168    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997181    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2226d542-31a5-4042-8aa7-43b49d0e5bac-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997194    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:06.997484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997278    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:06.997547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997531    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:06.997696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:06.997675    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token" pod "2226d542-31a5-4042-8aa7-43b49d0e5bac" (UID: "2226d542-31a5-4042-8aa7-43b49d0e5bac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:07.099243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.099206    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:07.099243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.099237    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:07.099243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.099247    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2226d542-31a5-4042-8aa7-43b49d0e5bac-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:07.213831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.213700    4944 generic.go:356] "Generic (PLEG): container finished" podID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerID="66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07" exitCode=0
Nov 03 19:15:07.213831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.213746    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"
Nov 03 19:15:07.213831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.213799    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" event={"ID":"2226d542-31a5-4042-8aa7-43b49d0e5bac","Type":"ContainerDied","Data":"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"}
Nov 03 19:15:07.214375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.213872    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t" event={"ID":"2226d542-31a5-4042-8aa7-43b49d0e5bac","Type":"ContainerDied","Data":"a7fc93946cc6460f9a6fdcc5b83151add9686acbc99b5ae27db1547d396d3411"}
Nov 03 19:15:07.214375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.213903    4944 scope.go:122] "RemoveContainer" containerID="66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"
Nov 03 19:15:07.229612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.229588    4944 scope.go:122] "RemoveContainer" containerID="66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"
Nov 03 19:15:07.229960 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:07.229937    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07\": container with ID starting with 66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07 not found: ID does not exist" containerID="66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"
Nov 03 19:15:07.230058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.229967    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07"} err="failed to get container status \"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07\": rpc error: code = NotFound desc = could not find container \"66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07\": container with ID starting with 66ec1a4cc0e19dc79627bf1c0bc0049f0ba5a0dc77216992afaf4afcb35ede07 not found: ID does not exist"
Nov 03 19:15:07.380926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.380891    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:15:07.386572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.386541    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-9jz8t"]
Nov 03 19:15:07.937002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:07.936964    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" path="/var/lib/kubelet/pods/2226d542-31a5-4042-8aa7-43b49d0e5bac/volumes"
Nov 03 19:15:10.816813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:10.804024    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:10.851727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:10.849494    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:11.088004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.087860    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:11.190372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.190331    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\") pod \"36d692a4-9aeb-463a-91a3-fcc56157ea48\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") "
Nov 03 19:15:11.190552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.190455    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\" (UniqueName: \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\") pod \"36d692a4-9aeb-463a-91a3-fcc56157ea48\" (UID: \"36d692a4-9aeb-463a-91a3-fcc56157ea48\") "
Nov 03 19:15:11.190741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.190718    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca" pod "36d692a4-9aeb-463a-91a3-fcc56157ea48" (UID: "36d692a4-9aeb-463a-91a3-fcc56157ea48"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:11.191516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.191491    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36d692a4-9aeb-463a-91a3-fcc56157ea48-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:11.193205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.192862    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq" pod "36d692a4-9aeb-463a-91a3-fcc56157ea48" (UID: "36d692a4-9aeb-463a-91a3-fcc56157ea48"). InnerVolumeSpecName "kube-api-access-l9bbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:11.252689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.252659    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"
Nov 03 19:15:11.252953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.252690    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg" event={"ID":"36d692a4-9aeb-463a-91a3-fcc56157ea48","Type":"ContainerDied","Data":"e5334b6801c7483f77a4b1d5fb2ec9e6bf5663601c6e2e89350724cb57f2f58c"}
Nov 03 19:15:11.253020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.252994    4944 scope.go:122] "RemoveContainer" containerID="eead4dbcf9b5fe4c30c7fdebb5635c05356f5a2aa45028f37889b7c1159b3543"
Nov 03 19:15:11.293443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.293406    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9bbq\" (UniqueName: \"kubernetes.io/projected/36d692a4-9aeb-463a-91a3-fcc56157ea48-kube-api-access-l9bbq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:11.410160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.410084    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:11.414426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.414396    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-5b012841-29895555-96sdg"]
Nov 03 19:15:11.938546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:11.938511    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" path="/var/lib/kubelet/pods/36d692a4-9aeb-463a-91a3-fcc56157ea48/volumes"
Nov 03 19:15:12.844050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:12.844010    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:13.102352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.102268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:13.365927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.365823    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:13.470296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.470250    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"]
Nov 03 19:15:13.520245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.520179    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-5b012841-cdjd7"]
Nov 03 19:15:13.538040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.538007    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:15:13.538240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.538217    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy" containerID="cri-o://6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b" gracePeriod=30
Nov 03 19:15:13.695866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.689248    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:13.938950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:13.938914    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20393455-57b1-4c20-ac21-682b66815215" path="/var/lib/kubelet/pods/20393455-57b1-4c20-ac21-682b66815215/volumes"
Nov 03 19:15:14.427997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:14.427963    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:14.933825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:14.930121    4944 scope.go:122] "RemoveContainer" containerID="e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"
Nov 03 19:15:14.933825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:14.931259    4944 scope.go:122] "RemoveContainer" containerID="4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"
Nov 03 19:15:14.933825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:14.932067    4944 scope.go:122] "RemoveContainer" containerID="fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"
Nov 03 19:15:15.260629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:15.260592    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:15.567778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:15.567745    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:15.927119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:15.927030    4944 scope.go:122] "RemoveContainer" containerID="4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"
Nov 03 19:15:15.927279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:15.927219    4944 scope.go:122] "RemoveContainer" containerID="d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab"
Nov 03 19:15:15.927378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:15.927357    4944 scope.go:122] "RemoveContainer" containerID="83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"
Nov 03 19:15:16.298220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.298195    4944 generic.go:356] "Generic (PLEG): container finished" podID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerID="fbde527c63196d699b32330cbd899f11d7ff15e466572184f01c697bd3c38d9c" exitCode=60
Nov 03 19:15:16.298328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.298267    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerDied","Data":"fbde527c63196d699b32330cbd899f11d7ff15e466572184f01c697bd3c38d9c"}
Nov 03 19:15:16.298390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.298330    4944 scope.go:122] "RemoveContainer" containerID="e22144e857290567442245275da6246757478a6380091d083af9f3a0f66af059"
Nov 03 19:15:16.298707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.298676    4944 scope.go:122] "RemoveContainer" containerID="fbde527c63196d699b32330cbd899f11d7ff15e466572184f01c697bd3c38d9c"
Nov 03 19:15:16.299087 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:16.299065    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w_odh-ai-gateway-infra(e7f10652-ddb3-4d46-b5f7-c9e4c26ff448)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448"
Nov 03 19:15:16.300296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.300275    4944 generic.go:356] "Generic (PLEG): container finished" podID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerID="e5bdd023bacbb7d170eb397252c90b523fc5c4c73c0731cbd1b72c0fb5ae4abf" exitCode=60
Nov 03 19:15:16.300382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.300345    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerDied","Data":"e5bdd023bacbb7d170eb397252c90b523fc5c4c73c0731cbd1b72c0fb5ae4abf"}
Nov 03 19:15:16.300760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.300742    4944 scope.go:122] "RemoveContainer" containerID="e5bdd023bacbb7d170eb397252c90b523fc5c4c73c0731cbd1b72c0fb5ae4abf"
Nov 03 19:15:16.301058 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:16.301033    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c_odh-ai-gateway-infra(ddd2bba4-2d87-4520-9b59-d9236e88c3df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df"
Nov 03 19:15:16.302616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.302596    4944 generic.go:356] "Generic (PLEG): container finished" podID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerID="ed5a803932932699008fcc5787256ccdda57f26781a64465a61064dbab7c97e6" exitCode=60
Nov 03 19:15:16.302712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.302666    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerDied","Data":"ed5a803932932699008fcc5787256ccdda57f26781a64465a61064dbab7c97e6"}
Nov 03 19:15:16.303024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.303009    4944 scope.go:122] "RemoveContainer" containerID="ed5a803932932699008fcc5787256ccdda57f26781a64465a61064dbab7c97e6"
Nov 03 19:15:16.303270 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:16.303252    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2_odh-ai-gateway-infra(74683d6c-0f1d-45ae-83e6-e73b726a3fe1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1"
Nov 03 19:15:16.395199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.395176    4944 scope.go:122] "RemoveContainer" containerID="fd5f5910b0edb77ad88f9c82772f80c60f524a7fd6c185772edfc45cc4d59ed7"
Nov 03 19:15:16.416379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.416350    4944 scope.go:122] "RemoveContainer" containerID="4b5677a86e31b438d0d3639c45ed2d94f5cc52a4ba1ef3a015e8563e8ef951fc"
Nov 03 19:15:16.814864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.795608    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:16.927122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.927089    4944 scope.go:122] "RemoveContainer" containerID="b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"
Nov 03 19:15:16.927341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:16.927319    4944 scope.go:122] "RemoveContainer" containerID="8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d"
Nov 03 19:15:17.173649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.173614    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:17.316353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.316317    4944 generic.go:356] "Generic (PLEG): container finished" podID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerID="790faed54b6559fd935271155f149b0e27a307b3e435e5d481d560c3c2fbcb57" exitCode=60
Nov 03 19:15:17.317013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.316547    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerDied","Data":"790faed54b6559fd935271155f149b0e27a307b3e435e5d481d560c3c2fbcb57"}
Nov 03 19:15:17.317163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.317062    4944 scope.go:122] "RemoveContainer" containerID="4bd08218d6c2e965832fb1cb18691e65fe233f70505284ea56e4c537c0f69e9e"
Nov 03 19:15:17.317650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.317629    4944 scope.go:122] "RemoveContainer" containerID="790faed54b6559fd935271155f149b0e27a307b3e435e5d481d560c3c2fbcb57"
Nov 03 19:15:17.318066 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:17.318041    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5_odh-ai-gateway-infra(10f9b167-ec6c-449f-b240-fa3072a1e2a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9"
Nov 03 19:15:17.321712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.321691    4944 generic.go:356] "Generic (PLEG): container finished" podID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerID="30e40102f2ac6f3bcbbc0e42c0ca20636786d67f3c8d85d6c38bc8bd9b4de2a9" exitCode=60
Nov 03 19:15:17.321826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.321762    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerDied","Data":"30e40102f2ac6f3bcbbc0e42c0ca20636786d67f3c8d85d6c38bc8bd9b4de2a9"}
Nov 03 19:15:17.322286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.322265    4944 scope.go:122] "RemoveContainer" containerID="30e40102f2ac6f3bcbbc0e42c0ca20636786d67f3c8d85d6c38bc8bd9b4de2a9"
Nov 03 19:15:17.322718 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:17.322692    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc_odh-ai-gateway-infra(773df7ae-91e4-4b53-94fd-d6f257e50cde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde"
Nov 03 19:15:17.328285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.328259    4944 generic.go:356] "Generic (PLEG): container finished" podID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerID="49de47c08fbffb45c28f4f823b8c31b6fdfc32901a9dd0af8364c18362095df6" exitCode=60
Nov 03 19:15:17.328378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.328300    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" event={"ID":"d6b1c257-9976-400d-b881-b6f71c959b0e","Type":"ContainerDied","Data":"49de47c08fbffb45c28f4f823b8c31b6fdfc32901a9dd0af8364c18362095df6"}
Nov 03 19:15:17.328739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.328721    4944 scope.go:122] "RemoveContainer" containerID="49de47c08fbffb45c28f4f823b8c31b6fdfc32901a9dd0af8364c18362095df6"
Nov 03 19:15:17.329287 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:17.329235    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw_odh-ai-gateway-infra(d6b1c257-9976-400d-b881-b6f71c959b0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e"
Nov 03 19:15:17.338925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.338895    4944 scope.go:122] "RemoveContainer" containerID="83d1ec89c50cfa78b2e6b951a21f9f7ba3def23bf20ddbe4557e40a408a03927"
Nov 03 19:15:17.407590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.407564    4944 scope.go:122] "RemoveContainer" containerID="d8f4ed677047a2664d1d9840ecae85096d569e9ec7640fcb157a035d8082d4ab"
Nov 03 19:15:17.447848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.447754    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:17.553760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.537413    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:17.666555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.659124    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:17.914380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:17.898693    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:18.088738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.088695    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\") pod \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") "
Nov 03 19:15:18.089306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.089283    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\" (UniqueName: \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\") pod \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\" (UID: \"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448\") "
Nov 03 19:15:18.093472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.093433    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca" pod "e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" (UID: "e7f10652-ddb3-4d46-b5f7-c9e4c26ff448"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.107989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.107956    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf" pod "e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" (UID: "e7f10652-ddb3-4d46-b5f7-c9e4c26ff448"). InnerVolumeSpecName "kube-api-access-97xgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.118076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.117968    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:18.211884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.196445    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97xgf\" (UniqueName: \"kubernetes.io/projected/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-kube-api-access-97xgf\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.212127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.212106    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.318346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.313858    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\") pod \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") "
Nov 03 19:15:18.318346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.314238    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\" (UniqueName: \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\") pod \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\" (UID: \"74683d6c-0f1d-45ae-83e6-e73b726a3fe1\") "
Nov 03 19:15:18.320279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.320000    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca" pod "74683d6c-0f1d-45ae-83e6-e73b726a3fe1" (UID: "74683d6c-0f1d-45ae-83e6-e73b726a3fe1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.336958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.336559    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:18.351429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.351353    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c" pod "74683d6c-0f1d-45ae-83e6-e73b726a3fe1" (UID: "74683d6c-0f1d-45ae-83e6-e73b726a3fe1"). InnerVolumeSpecName "kube-api-access-sr42c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.404147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.404110    4944 generic.go:356] "Generic (PLEG): container finished" podID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerID="41059c0f26ff508bcc2b4ca1d35357519225787b6ab801ad2537bfd51d92cb60" exitCode=60
Nov 03 19:15:18.410985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.408805    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerDied","Data":"41059c0f26ff508bcc2b4ca1d35357519225787b6ab801ad2537bfd51d92cb60"}
Nov 03 19:15:18.411270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.411251    4944 scope.go:122] "RemoveContainer" containerID="b7b043141c5bef3c933091e797208f9fb27836e49215bd8f691e981b79146ddd"
Nov 03 19:15:18.411903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.411883    4944 scope.go:122] "RemoveContainer" containerID="41059c0f26ff508bcc2b4ca1d35357519225787b6ab801ad2537bfd51d92cb60"
Nov 03 19:15:18.412516 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:18.412488    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l_odh-ai-gateway-infra(543790ca-2db0-4851-8cc0-20ebb3d99160)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160"
Nov 03 19:15:18.415582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.415558    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\") pod \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") "
Nov 03 19:15:18.416781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.416490    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\" (UniqueName: \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\") pod \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\" (UID: \"ddd2bba4-2d87-4520-9b59-d9236e88c3df\") "
Nov 03 19:15:18.417950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.417386    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr42c\" (UniqueName: \"kubernetes.io/projected/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-kube-api-access-sr42c\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.417950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.417411    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74683d6c-0f1d-45ae-83e6-e73b726a3fe1-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.419467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.419285    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca" pod "ddd2bba4-2d87-4520-9b59-d9236e88c3df" (UID: "ddd2bba4-2d87-4520-9b59-d9236e88c3df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.419624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.419604    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"
Nov 03 19:15:18.419836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.419726    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w" event={"ID":"e7f10652-ddb3-4d46-b5f7-c9e4c26ff448","Type":"ContainerDied","Data":"74120c19c5b78e420bc5c164d8aa4bd16637611363a3e298076ea082528a2234"}
Nov 03 19:15:18.420125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.420102    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9" pod "ddd2bba4-2d87-4520-9b59-d9236e88c3df" (UID: "ddd2bba4-2d87-4520-9b59-d9236e88c3df"). InnerVolumeSpecName "kube-api-access-w8cf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.421663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.421645    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"
Nov 03 19:15:18.422206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.421644    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c" event={"ID":"ddd2bba4-2d87-4520-9b59-d9236e88c3df","Type":"ContainerDied","Data":"0236da1c731a6a1e8447427e25292dcca0f8795a28c7ec181173d09ecb62dfe3"}
Nov 03 19:15:18.429411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.429382    4944 generic.go:356] "Generic (PLEG): container finished" podID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerID="02d03ee43e9d4039d39894dd07b920bb0cbaa00624eb43c086ad211092aeb6e3" exitCode=60
Nov 03 19:15:18.429528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.429455    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerDied","Data":"02d03ee43e9d4039d39894dd07b920bb0cbaa00624eb43c086ad211092aeb6e3"}
Nov 03 19:15:18.430067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.430043    4944 scope.go:122] "RemoveContainer" containerID="02d03ee43e9d4039d39894dd07b920bb0cbaa00624eb43c086ad211092aeb6e3"
Nov 03 19:15:18.430658 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:18.430630    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz_odh-ai-gateway-infra(3b1cdd76-fe72-43be-aa6f-b9674622c979)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979"
Nov 03 19:15:18.432997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.432975    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2" event={"ID":"74683d6c-0f1d-45ae-83e6-e73b726a3fe1","Type":"ContainerDied","Data":"5eaf51967f4d3568e72636e95d23202a9dd6a7c0bda7792a14e81fd90c08ca76"}
Nov 03 19:15:18.433130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.433016    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"
Nov 03 19:15:18.442878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.442772    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:18.450216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.450190    4944 scope.go:122] "RemoveContainer" containerID="fbde527c63196d699b32330cbd899f11d7ff15e466572184f01c697bd3c38d9c"
Nov 03 19:15:18.479616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.479585    4944 scope.go:122] "RemoveContainer" containerID="e5bdd023bacbb7d170eb397252c90b523fc5c4c73c0731cbd1b72c0fb5ae4abf"
Nov 03 19:15:18.505562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.505541    4944 scope.go:122] "RemoveContainer" containerID="8d7ac9e2047031412a6f541af52d173a1164300d62c4f8cc1c37291ada89bb4d"
Nov 03 19:15:18.519529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.519489    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8cf9\" (UniqueName: \"kubernetes.io/projected/ddd2bba4-2d87-4520-9b59-d9236e88c3df-kube-api-access-w8cf9\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.519529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.519528    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd2bba4-2d87-4520-9b59-d9236e88c3df-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.526243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.526221    4944 scope.go:122] "RemoveContainer" containerID="ed5a803932932699008fcc5787256ccdda57f26781a64465a61064dbab7c97e6"
Nov 03 19:15:18.557840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.557809    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:18.590886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.590855    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:18.595912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.595888    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-90a9bf-29895555-x5m6w"]
Nov 03 19:15:18.669455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.669427    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:18.709998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.709970    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:18.717872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.717831    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:18.722446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.722412    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-92596a-29895555-65gj2"]
Nov 03 19:15:18.724050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.723245    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\") pod \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") "
Nov 03 19:15:18.724050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.723513    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\" (UniqueName: \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\") pod \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\" (UID: \"10f9b167-ec6c-449f-b240-fa3072a1e2a9\") "
Nov 03 19:15:18.729147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.729118    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca" pod "10f9b167-ec6c-449f-b240-fa3072a1e2a9" (UID: "10f9b167-ec6c-449f-b240-fa3072a1e2a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.729498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.729462    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l" pod "10f9b167-ec6c-449f-b240-fa3072a1e2a9" (UID: "10f9b167-ec6c-449f-b240-fa3072a1e2a9"). InnerVolumeSpecName "kube-api-access-qx94l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.787439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.787406    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:18.856607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.856528    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\") pod \"773df7ae-91e4-4b53-94fd-d6f257e50cde\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") "
Nov 03 19:15:18.856782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.856689    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\" (UniqueName: \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\") pod \"773df7ae-91e4-4b53-94fd-d6f257e50cde\" (UID: \"773df7ae-91e4-4b53-94fd-d6f257e50cde\") "
Nov 03 19:15:18.857312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.857289    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx94l\" (UniqueName: \"kubernetes.io/projected/10f9b167-ec6c-449f-b240-fa3072a1e2a9-kube-api-access-qx94l\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.857312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.857313    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10f9b167-ec6c-449f-b240-fa3072a1e2a9-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.858016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.857973    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca" pod "773df7ae-91e4-4b53-94fd-d6f257e50cde" (UID: "773df7ae-91e4-4b53-94fd-d6f257e50cde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:18.862896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.862834    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:18.877249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.877177    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks" pod "773df7ae-91e4-4b53-94fd-d6f257e50cde" (UID: "773df7ae-91e4-4b53-94fd-d6f257e50cde"). InnerVolumeSpecName "kube-api-access-nmdks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:18.946164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.946113    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-4702dd-29895555-6cp7c"]
Nov 03 19:15:18.960945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.960909    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/773df7ae-91e4-4b53-94fd-d6f257e50cde-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.961159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.961141    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmdks\" (UniqueName: \"kubernetes.io/projected/773df7ae-91e4-4b53-94fd-d6f257e50cde-kube-api-access-nmdks\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:18.997206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:18.997141    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:19.067898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.067865    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:19.238419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.170842    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\") pod \"d6b1c257-9976-400d-b881-b6f71c959b0e\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") "
Nov 03 19:15:19.256837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.249058    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\" (UniqueName: \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\") pod \"d6b1c257-9976-400d-b881-b6f71c959b0e\" (UID: \"d6b1c257-9976-400d-b881-b6f71c959b0e\") "
Nov 03 19:15:19.256837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.208212    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca" pod "d6b1c257-9976-400d-b881-b6f71c959b0e" (UID: "d6b1c257-9976-400d-b881-b6f71c959b0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:19.359394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.359292    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b1c257-9976-400d-b881-b6f71c959b0e-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:19.462853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.460726    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:19.711138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.711095    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb" pod "d6b1c257-9976-400d-b881-b6f71c959b0e" (UID: "d6b1c257-9976-400d-b881-b6f71c959b0e"). InnerVolumeSpecName "kube-api-access-gswtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:19.743689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.743650    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw" event={"ID":"d6b1c257-9976-400d-b881-b6f71c959b0e","Type":"ContainerDied","Data":"7eef10363e1893b17285f50cae1be042a38bb089b46632b1238eb5ecfad45229"}
Nov 03 19:15:19.743985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.743964    4944 scope.go:122] "RemoveContainer" containerID="49de47c08fbffb45c28f4f823b8c31b6fdfc32901a9dd0af8364c18362095df6"
Nov 03 19:15:19.744243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.744221    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"
Nov 03 19:15:19.773218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.745024    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:15:19.873813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.856451    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:19.898814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:19.884955    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gswtb\" (UniqueName: \"kubernetes.io/projected/d6b1c257-9976-400d-b881-b6f71c959b0e-kube-api-access-gswtb\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.004688    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.004834    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.004914    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.004968    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.005023    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.005089    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.005211    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.005272    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.019465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.005385    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.024570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.020442    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"
Nov 03 19:15:20.051676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.051638    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.074054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.073959    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:20.113852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.086039    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "kube-api-access-ng8x6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:20.115999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.115946    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.129611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.117028    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.163721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.161683    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.181419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.181371    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.182172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.182145    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.182562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.182545    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") pod \"c9445652-7b21-4d8f-a179-6a5b0151de0e\" (UID: \"c9445652-7b21-4d8f-a179-6a5b0151de0e\") "
Nov 03 19:15:20.183782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.183764    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.267884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.267847    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9445652-7b21-4d8f-a179-6a5b0151de0e-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.295096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.295065    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8x6\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-kube-api-access-ng8x6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.295282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.295265    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.295383 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:20.182402    4944 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c9445652-7b21-4d8f-a179-6a5b0151de0e/volumes/kubernetes.io~empty-dir/workload-socket
Nov 03 19:15:20.307867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.307829    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.308057 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:20.182771    4944 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/c9445652-7b21-4d8f-a179-6a5b0151de0e/volumes/kubernetes.io~empty-dir/workload-certs
Nov 03 19:15:20.308164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.308144    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.318868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.316869    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" path="/var/lib/kubelet/pods/74683d6c-0f1d-45ae-83e6-e73b726a3fe1/volumes"
Nov 03 19:15:20.332810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.317720    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.339167    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.323315    4944 generic.go:356] "Generic (PLEG): container finished" podID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerID="6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b" exitCode=0
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.323380    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.319932    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" path="/var/lib/kubelet/pods/ddd2bba4-2d87-4520-9b59-d9236e88c3df/volumes"
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.342488    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" path="/var/lib/kubelet/pods/e7f10652-ddb3-4d46-b5f7-c9e4c26ff448/volumes"
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.420361    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5" event={"ID":"10f9b167-ec6c-449f-b240-fa3072a1e2a9","Type":"ContainerDied","Data":"cdab178e80097e0a7b20b068d5fa097c18cc091e872716eb3047ded0371a9457"}
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.420433    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc" event={"ID":"773df7ae-91e4-4b53-94fd-d6f257e50cde","Type":"ContainerDied","Data":"9f64ab53ef8724f48efc9cb875b415d041a4c00ced6a8b20f03012251fe26fba"}
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.420466    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerDied","Data":"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"}
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.420495    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc" event={"ID":"c9445652-7b21-4d8f-a179-6a5b0151de0e","Type":"ContainerDied","Data":"44449c38f348e70d48923e969e7245dc87d03ea0dcd20fd94d9fb5478729874e"}
Nov 03 19:15:20.491892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.420539    4944 scope.go:122] "RemoveContainer" containerID="790faed54b6559fd935271155f149b0e27a307b3e435e5d481d560c3c2fbcb57"
Nov 03 19:15:20.523782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.493675    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:20.572438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.572401    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.587446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.587419    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.627835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.627774    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.654666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.649682    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.661953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.661918    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.681946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.681911    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:20.682513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.682477    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": context deadline exceeded"
Nov 03 19:15:20.684475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.684450    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:20.709857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.684606    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:20.726857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.726830    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:20.733269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.733241    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token" pod "c9445652-7b21-4d8f-a179-6a5b0151de0e" (UID: "c9445652-7b21-4d8f-a179-6a5b0151de0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:20.841872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.834700    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:20.841872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.837470    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.841872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.837501    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.841872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.837519    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9445652-7b21-4d8f-a179-6a5b0151de0e-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:20.924983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.910276    4944 scope.go:122] "RemoveContainer" containerID="30e40102f2ac6f3bcbbc0e42c0ca20636786d67f3c8d85d6c38bc8bd9b4de2a9"
Nov 03 19:15:20.943562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.943527    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:20.980120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.979867    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.012837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:20.981138    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.031165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.024894    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.087812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.083719    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.179961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.167162    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.179961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.167558    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:15:21.302820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.237940    4944 scope.go:122] "RemoveContainer" containerID="6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"
Nov 03 19:15:21.393815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.393724    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Nov 03 19:15:21.398813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.394297    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:15:21.541155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.539916    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:21.669829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.667040    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.669829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.667132    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7" podUID="e1176c49-737c-4882-905e-799b99aa88fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.28:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.724833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.699060    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/ready\": context deadline exceeded"
Nov 03 19:15:21.752889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.714770    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:15:21.752889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.727841    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:15:21.779842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.754280    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded"
Nov 03 19:15:21.781698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.764130    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.797876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.784600    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:15:21.797876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.784665    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:15:21.797876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.766646    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 19:15:21.797876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.787683    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:15:21.797876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.796894    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.809722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.809522    4944 scope.go:122] "RemoveContainer" containerID="e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"
Nov 03 19:15:21.809722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.809670    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-144890-29895555-kt2pw"]
Nov 03 19:15:21.826942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.826726    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:21.919301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.919270    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:15:21.922514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.922490    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:15:21.954880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.954854    4944 scope.go:122] "RemoveContainer" containerID="6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"
Nov 03 19:15:21.965187 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:21.965138    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b\": container with ID starting with 6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b not found: ID does not exist" containerID="6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"
Nov 03 19:15:21.965435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.965386    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b"} err="failed to get container status \"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b\": rpc error: code = NotFound desc = could not find container \"6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b\": container with ID starting with 6d754e1ac57e8bfbc6391c564770c46aadf0260b519c41947b420dda710eb75b not found: ID does not exist"
Nov 03 19:15:21.965612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.965596    4944 scope.go:122] "RemoveContainer" containerID="e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"
Nov 03 19:15:21.966274 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:21.966191    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c\": container with ID starting with e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c not found: ID does not exist" containerID="e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"
Nov 03 19:15:21.966274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:21.966245    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c"} err="failed to get container status \"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c\": rpc error: code = NotFound desc = could not find container \"e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c\": container with ID starting with e0ea4b7097861ec98c5622bf5a673a0f16048601f6fe956ec8df12398c56538c not found: ID does not exist"
Nov 03 19:15:22.025122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.019262    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.100:8000/ready\": context deadline exceeded"
Nov 03 19:15:22.026439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.025502    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:22.097229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.097190    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" path="/var/lib/kubelet/pods/d6b1c257-9976-400d-b881-b6f71c959b0e/volumes"
Nov 03 19:15:22.097950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.097846    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:15:22.097950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.097920    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:15:22.098094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.098054    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:15:22.118818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.105438    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:15:22.144015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.143962    4944 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:15:22.156963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.156927    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:15:22.161205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.161163    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:22.162408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.162388    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:15:22.168639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.168608    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7"
Nov 03 19:15:22.168858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.168836    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:15:22.169001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.168980    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-98860e-29895555-46rt5"]
Nov 03 19:15:22.181093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.181059    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:22.184200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.183784    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:15:22.189877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.189831    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-81a044-29895555-g7chc"]
Nov 03 19:15:22.206808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.193965    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:22.206808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.196923    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:22.207055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.207037    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:15:22.256849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.256819    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5b012841-openshift-default-65b89d99bf-h82lc"]
Nov 03 19:15:22.368899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.368811    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\") pod \"3b1cdd76-fe72-43be-aa6f-b9674622c979\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") "
Nov 03 19:15:22.369255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.369211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca" pod "3b1cdd76-fe72-43be-aa6f-b9674622c979" (UID: "3b1cdd76-fe72-43be-aa6f-b9674622c979"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:22.369400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.369251    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\") pod \"543790ca-2db0-4851-8cc0-20ebb3d99160\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") "
Nov 03 19:15:22.369643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.369623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\" (UniqueName: \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\") pod \"543790ca-2db0-4851-8cc0-20ebb3d99160\" (UID: \"543790ca-2db0-4851-8cc0-20ebb3d99160\") "
Nov 03 19:15:22.369745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.369659    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca" pod "543790ca-2db0-4851-8cc0-20ebb3d99160" (UID: "543790ca-2db0-4851-8cc0-20ebb3d99160"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:22.369745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.369712    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\" (UniqueName: \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\") pod \"3b1cdd76-fe72-43be-aa6f-b9674622c979\" (UID: \"3b1cdd76-fe72-43be-aa6f-b9674622c979\") "
Nov 03 19:15:22.370510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.370487    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b1cdd76-fe72-43be-aa6f-b9674622c979-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:22.370596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.370513    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/543790ca-2db0-4851-8cc0-20ebb3d99160-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:22.372832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.372339    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n" pod "543790ca-2db0-4851-8cc0-20ebb3d99160" (UID: "543790ca-2db0-4851-8cc0-20ebb3d99160"). InnerVolumeSpecName "kube-api-access-4rh8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:22.373014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.372988    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q" pod "3b1cdd76-fe72-43be-aa6f-b9674622c979" (UID: "3b1cdd76-fe72-43be-aa6f-b9674622c979"). InnerVolumeSpecName "kube-api-access-hmn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:22.472102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.472067    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmn4q\" (UniqueName: \"kubernetes.io/projected/3b1cdd76-fe72-43be-aa6f-b9674622c979-kube-api-access-hmn4q\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:22.472245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:22.472103    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rh8n\" (UniqueName: \"kubernetes.io/projected/543790ca-2db0-4851-8cc0-20ebb3d99160-kube-api-access-4rh8n\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:23.195495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.195413    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"
Nov 03 19:15:23.195495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.195442    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l" event={"ID":"543790ca-2db0-4851-8cc0-20ebb3d99160","Type":"ContainerDied","Data":"868a819ca107aad5171ea32d7091c6d5c86f28106af9588b90c5a34f0efbee1f"}
Nov 03 19:15:23.196025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.195513    4944 scope.go:122] "RemoveContainer" containerID="41059c0f26ff508bcc2b4ca1d35357519225787b6ab801ad2537bfd51d92cb60"
Nov 03 19:15:23.198841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.198771    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"
Nov 03 19:15:23.198951 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.198839    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz" event={"ID":"3b1cdd76-fe72-43be-aa6f-b9674622c979","Type":"ContainerDied","Data":"98381b99a536db3ed4ef083e2ac022b773ec14da36b0aa794d01881127f025fc"}
Nov 03 19:15:23.239654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.239629    4944 scope.go:122] "RemoveContainer" containerID="02d03ee43e9d4039d39894dd07b920bb0cbaa00624eb43c086ad211092aeb6e3"
Nov 03 19:15:23.903362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.903329    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:23.927955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.907765    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-08baeb-29895555-lzhxz"]
Nov 03 19:15:23.985299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.985144    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" path="/var/lib/kubelet/pods/10f9b167-ec6c-449f-b240-fa3072a1e2a9/volumes"
Nov 03 19:15:23.985754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.985734    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" path="/var/lib/kubelet/pods/3b1cdd76-fe72-43be-aa6f-b9674622c979/volumes"
Nov 03 19:15:23.986218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.986201    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" path="/var/lib/kubelet/pods/773df7ae-91e4-4b53-94fd-d6f257e50cde/volumes"
Nov 03 19:15:23.986932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:23.986913    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" path="/var/lib/kubelet/pods/c9445652-7b21-4d8f-a179-6a5b0151de0e/volumes"
Nov 03 19:15:24.039412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:24.039377    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:24.043247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:24.043218    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fa2878-29895555-gjk6l"]
Nov 03 19:15:24.430815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:24.429433    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:24.678989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:24.678951    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:25.939646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:25.939610    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" path="/var/lib/kubelet/pods/543790ca-2db0-4851-8cc0-20ebb3d99160/volumes"
Nov 03 19:15:26.587325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:26.587295    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:26.850433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:26.850349    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:27.212608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:27.212521    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:27.505403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:27.505370    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:29.288219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.288188    4944 generic.go:356] "Generic (PLEG): container finished" podID="8d530c57-094d-4782-86d5-db5f87d9087f" containerID="19adb32c8dcc29686b815ff94f027dc5b345460b097f29ac5b89930b18687f99" exitCode=0
Nov 03 19:15:29.288679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.288263    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerDied","Data":"19adb32c8dcc29686b815ff94f027dc5b345460b097f29ac5b89930b18687f99"}
Nov 03 19:15:29.288679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.288671    4944 scope.go:122] "RemoveContainer" containerID="19adb32c8dcc29686b815ff94f027dc5b345460b097f29ac5b89930b18687f99"
Nov 03 19:15:29.290333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.290304    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:29.290425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.290369    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:29.290425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.290404    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:29.843379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:29.843339    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:30.145874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:30.145488    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:30.548742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:30.548701    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerStarted","Data":"ada854372cc8dd6b6a01a68d1480f6b8ae643cccff1ab94742a4309f669d54b9"}
Nov 03 19:15:30.947994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:30.947916    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:15:30.949458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:30.949432    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:15:30.949582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:30.949480    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 03 19:15:31.536993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.536957    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:15:31.537203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.537176    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" containerID="cri-o://fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7" gracePeriod=30
Nov 03 19:15:31.579712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.579681    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:15:31.580078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.579909    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" containerID="cri-o://057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea" gracePeriod=30
Nov 03 19:15:31.583887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.583863    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 19:15:31.584072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.584052    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy" containerID="cri-o://5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640" gracePeriod=30
Nov 03 19:15:31.829864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.826869    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:15:31.966918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.966771    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:15:31.967372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:31.967341    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 03 19:15:32.948065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:32.948032    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:15:32.948449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:32.948092    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 03 19:15:33.947890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:33.947857    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:15:33.948077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:33.947908    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 03 19:15:34.772724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.772686    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:15:34.773134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.772744    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:15:34.957948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.957912    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 03 19:15:34.958224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.958197    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 03 19:15:34.966969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.966932    4944 patch_prober.go:29] interesting pod/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:15:34.967225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:34.967197    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:15:35.005468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:35.005435    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:35.245846 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:35.245772    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:15:35.951566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:35.951539    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:15:35.952904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:35.952867    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:15:36.895246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.895209    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 19:15:36.944851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.944826    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:15:36.949489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.949437    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:15:36.968418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968393    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.968726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968440    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.968726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968609    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.968726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968676    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.968911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968730    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.968911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968748    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:36.968911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968773    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.968911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968883    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968913    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968925    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968941    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.968986    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969040    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969084    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969113    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.969184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969152    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969111    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969214    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969195    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969238    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969252    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969289    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969335    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969383    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969435    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969448    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969489    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969488    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969558    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.969607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969594    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969628    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969684    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969691    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969743    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\") pod \"ae40ddfb-5a05-4863-beae-5c07be9dd146\" (UID: \"ae40ddfb-5a05-4863-beae-5c07be9dd146\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969812    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\") pod \"2370d8f3-38ed-422a-8163-ea4684f20252\" (UID: \"2370d8f3-38ed-422a-8163-ea4684f20252\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969841    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\") pod \"6625ab85-53ff-4163-ada7-c5c04cb6747b\" (UID: \"6625ab85-53ff-4163-ada7-c5c04cb6747b\") "
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969925    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.970454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.969931    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970733    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970755    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae40ddfb-5a05-4863-beae-5c07be9dd146-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970770    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970785    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970818    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6625ab85-53ff-4163-ada7-c5c04cb6747b-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970831    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970846    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970861    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970875    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970892    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.970910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.970906    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:36.971571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.971111    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.971571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.971397    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.971694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.971596    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:36.971694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.971650    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.972475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.972211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "kube-api-access-gk4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.972475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.972247    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.972475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.972463    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "kube-api-access-vptxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.973293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.973262    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.974281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.973340    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:36.974281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.973568    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:15:36.974281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974269    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:36.974901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974389    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:15:36.974901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974403    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "kube-api-access-4v6w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.974901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974486    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token" pod "2370d8f3-38ed-422a-8163-ea4684f20252" (UID: "2370d8f3-38ed-422a-8163-ea4684f20252"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.974901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974560    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token" pod "ae40ddfb-5a05-4863-beae-5c07be9dd146" (UID: "ae40ddfb-5a05-4863-beae-5c07be9dd146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:15:36.975117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:36.974961    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy" pod "6625ab85-53ff-4163-ada7-c5c04cb6747b" (UID: "6625ab85-53ff-4163-ada7-c5c04cb6747b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:15:37.072093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072051    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072083    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072094    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072104    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072113    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2370d8f3-38ed-422a-8163-ea4684f20252-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072121    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072130    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vptxj\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-kube-api-access-vptxj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072142    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072154    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2370d8f3-38ed-422a-8163-ea4684f20252-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072166    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072178    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6625ab85-53ff-4163-ada7-c5c04cb6747b-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072188    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4k2\" (UniqueName: \"kubernetes.io/projected/6625ab85-53ff-4163-ada7-c5c04cb6747b-kube-api-access-gk4k2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072197    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v6w7\" (UniqueName: \"kubernetes.io/projected/2370d8f3-38ed-422a-8163-ea4684f20252-kube-api-access-4v6w7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072206    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae40ddfb-5a05-4863-beae-5c07be9dd146-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072215    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2370d8f3-38ed-422a-8163-ea4684f20252-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.072477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.072223    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae40ddfb-5a05-4863-beae-5c07be9dd146-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:15:37.607515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.607485    4944 generic.go:356] "Generic (PLEG): container finished" podID="2370d8f3-38ed-422a-8163-ea4684f20252" containerID="5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640" exitCode=0
Nov 03 19:15:37.607670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.607552    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"
Nov 03 19:15:37.607670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.607570    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerDied","Data":"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"}
Nov 03 19:15:37.607670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.607646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl" event={"ID":"2370d8f3-38ed-422a-8163-ea4684f20252","Type":"ContainerDied","Data":"23d1746face52d753c6e4bf6706fa44251066d9bb1153996a793ef91a56c016f"}
Nov 03 19:15:37.607670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.607669    4944 scope.go:122] "RemoveContainer" containerID="5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"
Nov 03 19:15:37.609294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.609271    4944 generic.go:356] "Generic (PLEG): container finished" podID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerID="fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7" exitCode=0
Nov 03 19:15:37.609413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.609322    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"
Nov 03 19:15:37.609413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.609346    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" event={"ID":"6625ab85-53ff-4163-ada7-c5c04cb6747b","Type":"ContainerDied","Data":"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"}
Nov 03 19:15:37.609521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.609421    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh" event={"ID":"6625ab85-53ff-4163-ada7-c5c04cb6747b","Type":"ContainerDied","Data":"3dc2eefd830800d365acfb56667f41e9422657bb0c123f20a7c886069a5207bc"}
Nov 03 19:15:37.610986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.610963    4944 generic.go:356] "Generic (PLEG): container finished" podID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerID="057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea" exitCode=0
Nov 03 19:15:37.611113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.611033    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"
Nov 03 19:15:37.611113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.611051    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" event={"ID":"ae40ddfb-5a05-4863-beae-5c07be9dd146","Type":"ContainerDied","Data":"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"}
Nov 03 19:15:37.611229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.611131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn" event={"ID":"ae40ddfb-5a05-4863-beae-5c07be9dd146","Type":"ContainerDied","Data":"6f849723f90aa6031be27b076ca621b8da006b21e41b23c51b14cfcedf27d40f"}
Nov 03 19:15:37.620580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.620430    4944 scope.go:122] "RemoveContainer" containerID="26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"
Nov 03 19:15:37.630629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.630612    4944 scope.go:122] "RemoveContainer" containerID="5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"
Nov 03 19:15:37.630910 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:37.630891    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640\": container with ID starting with 5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640 not found: ID does not exist" containerID="5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"
Nov 03 19:15:37.630973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.630925    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640"} err="failed to get container status \"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640\": rpc error: code = NotFound desc = could not find container \"5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640\": container with ID starting with 5e8e17330f5a175fd3a5b892813f18e7b76da583da4ed42dcc981bb0af076640 not found: ID does not exist"
Nov 03 19:15:37.630973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.630942    4944 scope.go:122] "RemoveContainer" containerID="26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"
Nov 03 19:15:37.631176 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:37.631158    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248\": container with ID starting with 26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248 not found: ID does not exist" containerID="26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"
Nov 03 19:15:37.631218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.631182    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248"} err="failed to get container status \"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248\": rpc error: code = NotFound desc = could not find container \"26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248\": container with ID starting with 26c43467b74dcac07aa15f0f7c8adebb1c99aaabcdb55ac7cd8a5a6fa7187248 not found: ID does not exist"
Nov 03 19:15:37.631218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.631200    4944 scope.go:122] "RemoveContainer" containerID="fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"
Nov 03 19:15:37.641104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.641087    4944 scope.go:122] "RemoveContainer" containerID="fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"
Nov 03 19:15:37.641367 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:37.641344    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7\": container with ID starting with fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7 not found: ID does not exist" containerID="fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"
Nov 03 19:15:37.641436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.641372    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7"} err="failed to get container status \"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7\": rpc error: code = NotFound desc = could not find container \"fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7\": container with ID starting with fd50bb86a63720b53c2dc2ff5d2aeb67f51567972b7ea44a2a4e326d7bec64f7 not found: ID does not exist"
Nov 03 19:15:37.641436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.641387    4944 scope.go:122] "RemoveContainer" containerID="057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"
Nov 03 19:15:37.651004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.650985    4944 scope.go:122] "RemoveContainer" containerID="057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"
Nov 03 19:15:37.651276 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:15:37.651256    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea\": container with ID starting with 057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea not found: ID does not exist" containerID="057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"
Nov 03 19:15:37.651339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.651285    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea"} err="failed to get container status \"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea\": rpc error: code = NotFound desc = could not find container \"057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea\": container with ID starting with 057bd0c91d8e9bd847f3ad57be72518efab6efbbe2bacee2cf8890fad7d699ea not found: ID does not exist"
Nov 03 19:15:37.763323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.763292    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 19:15:37.809360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.809326    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-56dxl"]
Nov 03 19:15:37.869965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.869870    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:15:37.873612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.873584    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-7tbmh"]
Nov 03 19:15:37.937075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.937042    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" path="/var/lib/kubelet/pods/2370d8f3-38ed-422a-8163-ea4684f20252/volumes"
Nov 03 19:15:37.937628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.937611    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" path="/var/lib/kubelet/pods/6625ab85-53ff-4163-ada7-c5c04cb6747b/volumes"
Nov 03 19:15:37.975836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.975800    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:15:37.980528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:37.980501    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-98860e-openshift-default-5c8578b4fc-9hvcn"]
Nov 03 19:15:39.929183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:39.929143    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:39.939488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:39.939454    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" path="/var/lib/kubelet/pods/ae40ddfb-5a05-4863-beae-5c07be9dd146/volumes"
Nov 03 19:15:40.947281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:40.947240    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:41.203899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:41.196272    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:43.476667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.476628    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:15:43.477573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477549    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477575    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477595    4944 cpu_manager.go:414] "containerMap: removing container" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477603    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477617    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477624    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477635    4944 cpu_manager.go:414] "containerMap: removing container" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477642    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477653    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477659    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477673    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.477684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477679    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477693    4944 cpu_manager.go:414] "containerMap: removing container" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477701    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477710    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477718    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477730    4944 cpu_manager.go:414] "containerMap: removing container" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477738    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477748    4944 cpu_manager.go:414] "containerMap: removing container" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477755    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477768    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477774    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477784    4944 cpu_manager.go:414] "containerMap: removing container" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477806    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477818    4944 cpu_manager.go:414] "containerMap: removing container" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477825    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477845    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477852    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477861    4944 cpu_manager.go:414] "containerMap: removing container" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477868    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477896    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477904    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477934    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477941    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477963    4944 cpu_manager.go:414] "containerMap: removing container" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477970    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477979    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477986    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.477998    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478005    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478015    4944 cpu_manager.go:414] "containerMap: removing container" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478022    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478035    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478042    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478051    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478058    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478243    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478257    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478268    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478279    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478292    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478301    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:15:43.478291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478313    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478322    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478331    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478340    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478351    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2226d542-31a5-4042-8aa7-43b49d0e5bac" containerName="istio-proxy"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478359    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478368    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478376    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478385    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478393    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae40ddfb-5a05-4863-beae-5c07be9dd146" containerName="istio-proxy"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478403    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478411    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478421    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478430    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478440    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478451    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478460    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478470    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6625ab85-53ff-4163-ada7-c5c04cb6747b" containerName="istio-proxy"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478478    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478489    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478496    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478504    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:15:43.480257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.478513    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:15:43.483311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.483291    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.532397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.532369    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d86d1cd0-openshift-default-dockercfg-hhczd\""
Nov 03 19:15:43.585134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585099    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:15:43.585429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585404    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4q8\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585438    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585463    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585587    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585652    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585736    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585900    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585936    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.585998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.585971    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687623    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4q8\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687658    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687680    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687708    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687751    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.687873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687808    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687879    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687905    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.687942    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.688245    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.688324    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.688356    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.688448    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.688569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.688543    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.690543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.690519    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.690875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.690852    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.745277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.745252    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.745479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.745455    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4q8\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\") pod \"e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") " pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:43.797293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:43.797260    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:44.225468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.225390    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:15:44.228995 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:15:44.228968    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee584a2_0005_4944_bf36_770d98f710de.slice/crio-246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08 WatchSource:0}: Error finding container 246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08: Status 404 returned error can't find the container with id 246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08
Nov 03 19:15:44.231593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.231562    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:44.231666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.231629    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:44.231666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.231659    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:15:44.703840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.700693    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" event={"ID":"cee584a2-0005-4944-bf36-770d98f710de","Type":"ContainerStarted","Data":"8e1b67a5a2e3701f36d5b3cf8870e8f9694011d5f260bbe53b09dc817ceacaaa"}
Nov 03 19:15:44.703840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.700771    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" event={"ID":"cee584a2-0005-4944-bf36-770d98f710de","Type":"ContainerStarted","Data":"246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08"}
Nov 03 19:15:44.712609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.712578    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:44.798397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.798359    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:44.820977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:44.820899    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podStartSLOduration=1.820877584 podStartE2EDuration="1.820877584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:15:44.815702773 +0000 UTC m=+1843.093056357" watchObservedRunningTime="2026-11-03 19:15:44.820877584 +0000 UTC m=+1843.098231169"
Nov 03 19:15:45.448398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:45.448367    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:45.799552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:45.799519    4944 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 19:15:45.799976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:45.799570    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 19:15:46.268953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:46.268893    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:46.798845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:46.798103    4944 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 19:15:46.798845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:46.798178    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 19:15:47.017252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:47.017195    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:47.354983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:47.354937    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:47.797944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:47.797905    4944 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 19:15:47.798132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:47.797972    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 19:15:48.798388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:48.798356    4944 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 19:15:48.798964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:48.798409    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 19:15:49.430852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:49.430747    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:49.797915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:49.797877    4944 patch_prober.go:29] interesting pod/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Nov 03 19:15:49.798100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:49.797937    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Nov 03 19:15:50.844340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:50.844311    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:50.844819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:50.844683    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:50.845974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:50.845950    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:15:51.042045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:51.042012    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:51.299356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:51.299321    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:54.334749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:54.334698    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:54.583676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:54.583632    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:55.800160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:55.800079    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:15:59.651594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:15:59.651553    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:03.898274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.898241    4944 scope.go:122] "RemoveContainer" containerID="296b5cc513e1d786eeccc1f21a0c8fb4191e34310d857566e469921db90f9536"
Nov 03 19:16:03.909390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.909369    4944 scope.go:122] "RemoveContainer" containerID="edd4d0ef5d64f60208fb691e5bd8520aab08800359b05955ca9e6f11890993a1"
Nov 03 19:16:03.919082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.919055    4944 scope.go:122] "RemoveContainer" containerID="d1fd2895660a2cf6b6046b838e696f5cde7e456c392d6cb100b26d08f5a7b9cb"
Nov 03 19:16:03.930132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.930104    4944 scope.go:122] "RemoveContainer" containerID="252c42230db3064a08ee61d3fa6bca55ff0033050ec387b8bc33bdfa681cb002"
Nov 03 19:16:03.939853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.939836    4944 scope.go:122] "RemoveContainer" containerID="db3cb703ba34c8fc73cd9d8bffcbdc0959138b0d95f210b5d6298b041bb09732"
Nov 03 19:16:03.953115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.952884    4944 scope.go:122] "RemoveContainer" containerID="985a4042cb16fcc06d8823d51d5b9a4505d070ff03cb61465b719e4c10282dbc"
Nov 03 19:16:03.965496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:03.965467    4944 scope.go:122] "RemoveContainer" containerID="f271c108125d6dd4857b604ad48474030cbf347c861ee6a4f3c5214ea8aee3b1"
Nov 03 19:16:04.653864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:04.653822    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:05.225465 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:16:05.225430    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:16:09.615670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:09.615629    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:12.451999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:12.451965    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:12.718022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:12.717946    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:13.175806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:13.167690    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:13.519592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:13.519555    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:14.446343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:14.446306    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:17.365394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:17.365359    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:17.619003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:17.618908    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:19.283256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:19.283223    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:20.713855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:20.689063    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:20.969503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:20.969388    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:23.279604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:23.279570    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:23.587545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:23.587457    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:24.296022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:24.295954    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:27.529230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:27.529196    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:29.423929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:29.423899    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:29.681776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:29.681684    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:29.927020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:29.926979    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:32.047338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:32.047301    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"]
Nov 03 19:16:32.093136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:32.093099    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1452626f-gf9bw"]
Nov 03 19:16:33.938118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:33.938081    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244d362e-a6d0-4efb-ac35-7a4d2f80e968" path="/var/lib/kubelet/pods/244d362e-a6d0-4efb-ac35-7a4d2f80e968/volumes"
Nov 03 19:16:34.305529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:34.305496    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:34.549873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:34.549835    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:35.080923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:35.080881    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:35.227310 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:16:35.227273    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:16:35.955454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:35.955423    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:36.217957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:36.217861    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:39.398895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:39.398829    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:40.719527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:40.719487    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:41.070822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:41.070780    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:43.989312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:43.989275    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:44.236109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.236063    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:44.424352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.424321    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:16:44.425127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425109    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425129    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b1c257-9976-400d-b881-b6f71c959b0e" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425148    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425153    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2370d8f3-38ed-422a-8163-ea4684f20252" containerName="istio-proxy"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425160    4944 cpu_manager.go:414] "containerMap: removing container" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425165    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543790ca-2db0-4851-8cc0-20ebb3d99160" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425176    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425180    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:16:44.425185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425187    4944 cpu_manager.go:414] "containerMap: removing container" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425192    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425202    4944 cpu_manager.go:414] "containerMap: removing container" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425207    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425219    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425224    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9445652-7b21-4d8f-a179-6a5b0151de0e" containerName="istio-proxy"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425236    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425240    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd2bba4-2d87-4520-9b59-d9236e88c3df" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425261    4944 cpu_manager.go:414] "containerMap: removing container" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425265    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773df7ae-91e4-4b53-94fd-d6f257e50cde" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425276    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425280    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1cdd76-fe72-43be-aa6f-b9674622c979" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425388    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="74683d6c-0f1d-45ae-83e6-e73b726a3fe1" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425398    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f9b167-ec6c-449f-b240-fa3072a1e2a9" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425405    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="36d692a4-9aeb-463a-91a3-fcc56157ea48" containerName="cleanup"
Nov 03 19:16:44.425438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.425414    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f10652-ddb3-4d46-b5f7-c9e4c26ff448" containerName="cleanup"
Nov 03 19:16:44.437614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.437582    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.488987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.488951    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:16:44.525614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.525568    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1cc3f2\""
Nov 03 19:16:44.526061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.526000    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1cc3f2-dockercfg-pfknb\""
Nov 03 19:16:44.607227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.607181    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.607499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.607483    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctmc\" (UniqueName: \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.686065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.685965    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:44.709317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.709275    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctmc\" (UniqueName: \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.709596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.709573    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.710584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.710555    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.768871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.768834    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctmc\" (UniqueName: \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\") pod \"payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") " pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:44.934737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:44.934696    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:45.055426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:45.055396    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:45.184405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:45.183100    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:45.441387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:45.441302    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:16:45.446734 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:16:45.446680    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ca584f_24ea_49b6_9c0d_04d9d2150edd.slice/crio-38788e848e967cf52352dda5bd1e557084d36556d960bf5404cc22256939982b WatchSource:0}: Error finding container 38788e848e967cf52352dda5bd1e557084d36556d960bf5404cc22256939982b: Status 404 returned error can't find the container with id 38788e848e967cf52352dda5bd1e557084d36556d960bf5404cc22256939982b
Nov 03 19:16:46.218813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:46.218640    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" event={"ID":"b9ca584f-24ea-49b6-9c0d-04d9d2150edd","Type":"ContainerStarted","Data":"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"}
Nov 03 19:16:46.218813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:46.218717    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" event={"ID":"b9ca584f-24ea-49b6-9c0d-04d9d2150edd","Type":"ContainerStarted","Data":"38788e848e967cf52352dda5bd1e557084d36556d960bf5404cc22256939982b"}
Nov 03 19:16:46.218813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:46.218749    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:50.068837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:50.068780    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:50.880461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:50.880432    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:54.668800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:54.668749    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:16:57.310827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:57.310771    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:16:57.419928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:57.419872    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" podStartSLOduration=13.419855897 podStartE2EDuration="13.419855897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:16:46.331076515 +0000 UTC m=+1904.608430103" watchObservedRunningTime="2026-11-03 19:16:57.419855897 +0000 UTC m=+1915.697209481"
Nov 03 19:16:59.275270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:16:59.275226    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:00.377121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.377083    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"]
Nov 03 19:17:00.381729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.381711    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.435812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.435769    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"]
Nov 03 19:17:00.477540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.477509    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:17:00.477749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.477592    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:17:00.477850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.477592    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:17:00.477850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.477688    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:17:00.493604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.493576    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t7q6\" (UniqueName: \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.493782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.493761    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.493883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.493873    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.594760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.594719    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t7q6\" (UniqueName: \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.594989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.594966    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.595069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.595055    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.595605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.595577    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.597539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.597518    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.654219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.654156    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t7q6\" (UniqueName: \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:00.695848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:00.695825    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:01.069970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:01.069935    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"]
Nov 03 19:17:01.074721 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:17:01.074694    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ed8f96_8da1_4dfa_9b07_819ee5273af6.slice/crio-3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887 WatchSource:0}: Error finding container 3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887: Status 404 returned error can't find the container with id 3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887
Nov 03 19:17:01.412825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:01.412742    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h" event={"ID":"a3ed8f96-8da1-4dfa-9b07-819ee5273af6","Type":"ContainerStarted","Data":"3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887"}
Nov 03 19:17:02.423715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:02.423600    4944 generic.go:356] "Generic (PLEG): container finished" podID="a3ed8f96-8da1-4dfa-9b07-819ee5273af6" containerID="b9fe4d76635c53ee515a41ae66334221ded356109e4ee85e7dbe40760fa8c03b" exitCode=0
Nov 03 19:17:02.423715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:02.423684    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h" event={"ID":"a3ed8f96-8da1-4dfa-9b07-819ee5273af6","Type":"ContainerDied","Data":"b9fe4d76635c53ee515a41ae66334221ded356109e4ee85e7dbe40760fa8c03b"}
Nov 03 19:17:03.611486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.611460    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:03.739393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.739333    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\") pod \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") "
Nov 03 19:17:03.739565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.739434    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\" (UniqueName: \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\") pod \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") "
Nov 03 19:17:03.739565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.739541    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\") pod \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\" (UID: \"a3ed8f96-8da1-4dfa-9b07-819ee5273af6\") "
Nov 03 19:17:03.739888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.739861    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca" pod "a3ed8f96-8da1-4dfa-9b07-819ee5273af6" (UID: "a3ed8f96-8da1-4dfa-9b07-819ee5273af6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:17:03.740150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.740124    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:03.741618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.741596    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca" pod "a3ed8f96-8da1-4dfa-9b07-819ee5273af6" (UID: "a3ed8f96-8da1-4dfa-9b07-819ee5273af6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:17:03.741701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.741672    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6" pod "a3ed8f96-8da1-4dfa-9b07-819ee5273af6" (UID: "a3ed8f96-8da1-4dfa-9b07-819ee5273af6"). InnerVolumeSpecName "kube-api-access-4t7q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:17:03.841371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.841337    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:03.841371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:03.841371    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t7q6\" (UniqueName: \"kubernetes.io/projected/a3ed8f96-8da1-4dfa-9b07-819ee5273af6-kube-api-access-4t7q6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:04.068855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:04.068665    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:04.441075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:04.441000    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"
Nov 03 19:17:04.441075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:04.441019    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h" event={"ID":"a3ed8f96-8da1-4dfa-9b07-819ee5273af6","Type":"ContainerDied","Data":"3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887"}
Nov 03 19:17:04.441075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:04.441067    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa4ee6ed6d87067ce0a6ef8eafa76abba6524893e07de24c6728000892a2887"
Nov 03 19:17:05.225296 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:17:05.225263    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:17:08.618323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:08.618283    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:08.872316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:08.872219    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:09.121814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:09.121764    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:12.020760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:12.020671    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:17:12.121785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:12.121748    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:17:12.474207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:12.474120    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:12.716719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:12.716674    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:13.511813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:13.504357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:17.020995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:17.020949    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing" containerID="cri-o://ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046" gracePeriod=25
Nov 03 19:17:17.123096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:17.123034    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing" containerID="cri-o://8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34" gracePeriod=25
Nov 03 19:17:18.081915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:18.081872    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:18.958372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:18.958334    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:17:18.958556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:18.958393    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:17:22.915654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:22.915614    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:27.389356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:27.389318    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:27.954396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:27.954357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:28.958351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:28.958313    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:17:28.958754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:28.958325    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:17:32.257207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:32.257170    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:35.225297 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:17:35.225259    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:17:37.072567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.072528    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:37.315984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.315956    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:17:37.391303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.391273    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\" (UniqueName: \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\") pod \"4972333b-64e8-42cf-8420-c3caa5944b01\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") "
Nov 03 19:17:37.391500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.391484    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\") pod \"4972333b-64e8-42cf-8420-c3caa5944b01\" (UID: \"4972333b-64e8-42cf-8420-c3caa5944b01\") "
Nov 03 19:17:37.391910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.391882    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume" pod "4972333b-64e8-42cf-8420-c3caa5944b01" (UID: "4972333b-64e8-42cf-8420-c3caa5944b01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:17:37.392195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.392177    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4972333b-64e8-42cf-8420-c3caa5944b01-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:37.393468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.393445    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc" pod "4972333b-64e8-42cf-8420-c3caa5944b01" (UID: "4972333b-64e8-42cf-8420-c3caa5944b01"). InnerVolumeSpecName "kube-api-access-6l7jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:17:37.405681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.405659    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:17:37.492986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.492952    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\") pod \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") "
Nov 03 19:17:37.493160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.493045    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\" (UniqueName: \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\") pod \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\" (UID: \"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6\") "
Nov 03 19:17:37.493328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.493306    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume" pod "d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" (UID: "d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:17:37.493638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.493611    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:37.493638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.493638    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7jc\" (UniqueName: \"kubernetes.io/projected/4972333b-64e8-42cf-8420-c3caa5944b01-kube-api-access-6l7jc\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:37.495113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.495091    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq" pod "d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" (UID: "d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6"). InnerVolumeSpecName "kube-api-access-9qxzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:17:37.595159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.595083    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qxzq\" (UniqueName: \"kubernetes.io/projected/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6-kube-api-access-9qxzq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:37.740956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.740916    4944 generic.go:356] "Generic (PLEG): container finished" podID="4972333b-64e8-42cf-8420-c3caa5944b01" containerID="ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046" exitCode=0
Nov 03 19:17:37.741128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.740981    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"
Nov 03 19:17:37.741128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.740994    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" event={"ID":"4972333b-64e8-42cf-8420-c3caa5944b01","Type":"ContainerDied","Data":"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"}
Nov 03 19:17:37.741128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.741074    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v" event={"ID":"4972333b-64e8-42cf-8420-c3caa5944b01","Type":"ContainerDied","Data":"d8f976eec8092edb1211aecf07b7f2c3cb7398f2a07fa8ab001b639d3ecec67f"}
Nov 03 19:17:37.741128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.741096    4944 scope.go:122] "RemoveContainer" containerID="ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"
Nov 03 19:17:37.742731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.742710    4944 generic.go:356] "Generic (PLEG): container finished" podID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerID="8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34" exitCode=0
Nov 03 19:17:37.742879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.742774    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"
Nov 03 19:17:37.742879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.742799    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" event={"ID":"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6","Type":"ContainerDied","Data":"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"}
Nov 03 19:17:37.742879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.742837    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf" event={"ID":"d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6","Type":"ContainerDied","Data":"37500783b59d7c36bed0ed479a2d0678d792e43dbc2a34b6db1eec834eb3fac7"}
Nov 03 19:17:37.756452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.756411    4944 scope.go:122] "RemoveContainer" containerID="ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"
Nov 03 19:17:37.757847 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:17:37.757815    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046\": container with ID starting with ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046 not found: ID does not exist" containerID="ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"
Nov 03 19:17:37.757942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.757858    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046"} err="failed to get container status \"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046\": rpc error: code = NotFound desc = could not find container \"ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046\": container with ID starting with ba86c8f45d8dc1b0c4a57f153befe540d018c4cecf771a7e0eb4e00bb37e6046 not found: ID does not exist"
Nov 03 19:17:37.757942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.757883    4944 scope.go:122] "RemoveContainer" containerID="8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"
Nov 03 19:17:37.768226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.768199    4944 scope.go:122] "RemoveContainer" containerID="8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"
Nov 03 19:17:37.768485 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:17:37.768462    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34\": container with ID starting with 8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34 not found: ID does not exist" containerID="8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"
Nov 03 19:17:37.768550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.768496    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34"} err="failed to get container status \"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34\": rpc error: code = NotFound desc = could not find container \"8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34\": container with ID starting with 8d1f8abd7cac8d24d0939b1b351f6915fca5a7ef487a57bb1a0e31f96e84ef34 not found: ID does not exist"
Nov 03 19:17:37.899363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.899278    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:17:37.903223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.903194    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d86d1cd0-6b97d4bcb7-fplxf"]
Nov 03 19:17:37.936622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:37.936586    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" path="/var/lib/kubelet/pods/d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6/volumes"
Nov 03 19:17:38.003203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:38.003171    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:17:38.007759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:38.007730    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d86d1cd0-5cf5dc6748-srf2v"]
Nov 03 19:17:39.936322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:39.936284    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" path="/var/lib/kubelet/pods/4972333b-64e8-42cf-8420-c3caa5944b01/volumes"
Nov 03 19:17:41.622364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:41.622323    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:41.868697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:41.868655    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:42.299411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:42.299384    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:45.208827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:45.208779    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:17:45.209348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:45.208998    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy" containerID="cri-o://27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64" gracePeriod=30
Nov 03 19:17:46.991849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:46.991811    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:50.502085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.502055    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:17:50.627411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627376    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627467    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627521    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627563    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627642    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627715    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627757    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.627902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627760    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:17:50.628010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627977    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.628106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628078    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\") pod \"7c44d648-90d2-4f52-8008-ff18d660ef15\" (UID: \"7c44d648-90d2-4f52-8008-ff18d660ef15\") "
Nov 03 19:17:50.628168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.627975    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:17:50.628168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628098    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:17:50.628277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628176    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:17:50.628336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628305    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:17:50.628962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628937    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.628962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628967    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.629142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628983    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c44d648-90d2-4f52-8008-ff18d660ef15-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.629142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.628997    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.629142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.629010    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.629692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.629664    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:17:50.629987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.629965    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:17:50.629987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.629976    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "kube-api-access-scxlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:17:50.630105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.630080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo" pod "7c44d648-90d2-4f52-8008-ff18d660ef15" (UID: "7c44d648-90d2-4f52-8008-ff18d660ef15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:17:50.730749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.730664    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.730749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.730706    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.730749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.730725    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scxlj\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-kube-api-access-scxlj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.730749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.730741    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c44d648-90d2-4f52-8008-ff18d660ef15-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:17:50.864248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.864212    4944 generic.go:356] "Generic (PLEG): container finished" podID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerID="27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64" exitCode=0
Nov 03 19:17:50.864439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.864263    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"
Nov 03 19:17:50.864439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.864301    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" event={"ID":"7c44d648-90d2-4f52-8008-ff18d660ef15","Type":"ContainerDied","Data":"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"}
Nov 03 19:17:50.864439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.864357    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8" event={"ID":"7c44d648-90d2-4f52-8008-ff18d660ef15","Type":"ContainerDied","Data":"1389d22e5355cbd8ca229cc4b5dba4cd501db00a609cc2b7cb745fb4d4330eeb"}
Nov 03 19:17:50.864598 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.864432    4944 scope.go:122] "RemoveContainer" containerID="27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"
Nov 03 19:17:50.876080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.876065    4944 scope.go:122] "RemoveContainer" containerID="27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"
Nov 03 19:17:50.876333 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:17:50.876318    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64\": container with ID starting with 27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64 not found: ID does not exist" containerID="27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"
Nov 03 19:17:50.876390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:50.876340    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64"} err="failed to get container status \"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64\": rpc error: code = NotFound desc = could not find container \"27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64\": container with ID starting with 27beb4529e52e3c9a6aba58bf43037653fd7a29791d51e8ef99e040d3290bb64 not found: ID does not exist"
Nov 03 19:17:51.021263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:51.021229    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:17:51.024917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:51.024894    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-f2mw8"]
Nov 03 19:17:51.728045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:51.728004    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:17:51.937321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:51.937289    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" path="/var/lib/kubelet/pods/7c44d648-90d2-4f52-8008-ff18d660ef15/volumes"
Nov 03 19:17:54.047534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:54.047501    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"]
Nov 03 19:17:54.093328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:54.093291    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7afd59d-scslm"]
Nov 03 19:17:55.937504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:55.937467    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b3d180-c4a9-4700-a8e6-6e98b947dd13" path="/var/lib/kubelet/pods/c7b3d180-c4a9-4700-a8e6-6e98b947dd13/volumes"
Nov 03 19:17:56.209742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:17:56.209648    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:00.997051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:00.997015    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:03.090968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:03.090932    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:18:03.091390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:03.091148    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" containerID="cri-o://fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4" gracePeriod=30
Nov 03 19:18:03.382683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:03.382599    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:03.912199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:03.912157    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:04.113222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:04.113188    4944 scope.go:122] "RemoveContainer" containerID="ea1e6a20d20077abe010be988249d050a5c342867915820722d247eeb8a014be"
Nov 03 19:18:04.126080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:04.126056    4944 scope.go:122] "RemoveContainer" containerID="70e4512b7b49eff4417034619721b5b7a53a4a33793b2908ad09799b3254c331"
Nov 03 19:18:04.136103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:04.136080    4944 scope.go:122] "RemoveContainer" containerID="d9807aa27bb8a4ac8f5f21752d13d5c3e63be2f34d0468904095656aba872aa8"
Nov 03 19:18:04.145429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:04.145404    4944 scope.go:122] "RemoveContainer" containerID="62895844b5fe7e37adf705c4a862a0a7e11fa24f6917130682f395d011ed8ce9"
Nov 03 19:18:04.156578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:04.156555    4944 scope.go:122] "RemoveContainer" containerID="f7047955ff2848e98f3997804d307f60b88fee6f331f24421d701442aa6b2d2c"
Nov 03 19:18:05.225920 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:18:05.225885    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:18:05.688999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:05.688953    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:05.943709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:05.943612    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:06.239041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:06.238964    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:07.195619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:07.195583    4944 patch_prober.go:29] interesting pod/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:18:07.195812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:07.195647    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:18:08.370371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.370307    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:18:08.525569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525536    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.525740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525587    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.525740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525624    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.525740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525652    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.525740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525733    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.526037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525850    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.526037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525946    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.526037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525987    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.526194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.526077    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:08.526194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.525995    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:08.526299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.526264    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:08.526353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.526325    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:08.526466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.526446    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:08.526539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.526460    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\") pod \"326443af-af71-4f81-979e-72503ca4d9b3\" (UID: \"326443af-af71-4f81-979e-72503ca4d9b3\") "
Nov 03 19:18:08.527190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.527160    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/326443af-af71-4f81-979e-72503ca4d9b3-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.527190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.527181    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.527391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.527196    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.527391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.527209    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.527391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.527225    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.528746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.528705    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "kube-api-access-nx8x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:08.529681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.529646    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:18:08.529804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.529764    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:08.529875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.529772    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token" pod "326443af-af71-4f81-979e-72503ca4d9b3" (UID: "326443af-af71-4f81-979e-72503ca4d9b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:08.628606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.628532    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/326443af-af71-4f81-979e-72503ca4d9b3-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.628606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.628565    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.628606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.628575    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/326443af-af71-4f81-979e-72503ca4d9b3-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:08.628606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:08.628586    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8x8\" (UniqueName: \"kubernetes.io/projected/326443af-af71-4f81-979e-72503ca4d9b3-kube-api-access-nx8x8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:09.016209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.016177    4944 generic.go:356] "Generic (PLEG): container finished" podID="326443af-af71-4f81-979e-72503ca4d9b3" containerID="fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4" exitCode=0
Nov 03 19:18:09.016387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.016223    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"
Nov 03 19:18:09.016387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.016289    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" event={"ID":"326443af-af71-4f81-979e-72503ca4d9b3","Type":"ContainerDied","Data":"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"}
Nov 03 19:18:09.016387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.016348    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6" event={"ID":"326443af-af71-4f81-979e-72503ca4d9b3","Type":"ContainerDied","Data":"a7b91f08155d4d811f16891590292947b62620d11fff066e4c1ded1378c79cee"}
Nov 03 19:18:09.016387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.016373    4944 scope.go:122] "RemoveContainer" containerID="fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"
Nov 03 19:18:09.031009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.030980    4944 scope.go:122] "RemoveContainer" containerID="fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"
Nov 03 19:18:09.031323 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:18:09.031300    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4\": container with ID starting with fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4 not found: ID does not exist" containerID="fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"
Nov 03 19:18:09.031393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.031335    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4"} err="failed to get container status \"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4\": rpc error: code = NotFound desc = could not find container \"fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4\": container with ID starting with fcf23752c1b6195fae709e97939b9d4baf38f2fe71a7466015fdf4989ea4c2a4 not found: ID does not exist"
Nov 03 19:18:09.192906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.189732    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:18:09.219231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.219191    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-wj2x6"]
Nov 03 19:18:09.834980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.834943    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:09.938284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:09.938252    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326443af-af71-4f81-979e-72503ca4d9b3" path="/var/lib/kubelet/pods/326443af-af71-4f81-979e-72503ca4d9b3/volumes"
Nov 03 19:18:10.087121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:10.087014    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:11.374759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:11.374722    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:14.340128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:14.340066    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:14.586715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:14.586658    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:15.535733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:15.535699    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:16.281974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:16.281564    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:16.771628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:16.771597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:17.022897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:17.022782    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:17.424321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:17.424243    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:17.670123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:17.670081    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:20.270151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:20.270117    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:20.517305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:20.517244    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:21.909574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:21.909540    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:25.414543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.414508    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"]
Nov 03 19:18:25.415240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415224    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing"
Nov 03 19:18:25.415286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415243    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing"
Nov 03 19:18:25.415286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415259    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a3ed8f96-8da1-4dfa-9b07-819ee5273af6" containerName="revoke-keys"
Nov 03 19:18:25.415286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415264    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ed8f96-8da1-4dfa-9b07-819ee5273af6" containerName="revoke-keys"
Nov 03 19:18:25.415286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415281    4944 cpu_manager.go:414] "containerMap: removing container" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy"
Nov 03 19:18:25.415286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415287    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415296    4944 cpu_manager.go:414] "containerMap: removing container" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415301    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415309    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415314    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415426    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ed8f96-8da1-4dfa-9b07-819ee5273af6" containerName="revoke-keys"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415436    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="326443af-af71-4f81-979e-72503ca4d9b3" containerName="istio-proxy"
Nov 03 19:18:25.415440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415442    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="4972333b-64e8-42cf-8420-c3caa5944b01" containerName="payload-processing"
Nov 03 19:18:25.415652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415448    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c44d648-90d2-4f52-8008-ff18d660ef15" containerName="istio-proxy"
Nov 03 19:18:25.415652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.415462    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d6ef4f-07d3-4ab4-bac3-2cfa745470f6" containerName="payload-pre-processing"
Nov 03 19:18:25.419046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.419027    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.463529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.463501    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.463840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.463816    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcmn\" (UniqueName: \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.463896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.463857    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.472877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.472843    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"]
Nov 03 19:18:25.514267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.514237    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:18:25.514267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.514246    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:18:25.514470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.514275    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:18:25.514470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.514302    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:18:25.565114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.565083    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcmn\" (UniqueName: \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.565114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.565115    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.565321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.565161    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.565920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.565900    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.567492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.567473    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.623270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.623233    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcmn\" (UniqueName: \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:25.733485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:25.733401    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:26.108672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:26.108637    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"]
Nov 03 19:18:26.113303 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:18:26.113274    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc869c3e1_4241_46fc_bb31_e490e72d7805.slice/crio-941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8 WatchSource:0}: Error finding container 941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8: Status 404 returned error can't find the container with id 941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8
Nov 03 19:18:26.156491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:26.156462    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx" event={"ID":"c869c3e1-4241-46fc-bb31-e490e72d7805","Type":"ContainerStarted","Data":"941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8"}
Nov 03 19:18:26.882845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:26.882771    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:27.164986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:27.164898    4944 generic.go:356] "Generic (PLEG): container finished" podID="c869c3e1-4241-46fc-bb31-e490e72d7805" containerID="4a5c0af59fc8c3f4f64a1789d0b5b3813e975961fe7e7329e7d089e9e568e2c0" exitCode=0
Nov 03 19:18:27.165150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:27.164978    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx" event={"ID":"c869c3e1-4241-46fc-bb31-e490e72d7805","Type":"ContainerDied","Data":"4a5c0af59fc8c3f4f64a1789d0b5b3813e975961fe7e7329e7d089e9e568e2c0"}
Nov 03 19:18:28.318117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.318089    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:28.342122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.342096    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:28.405400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.405365    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\" (UniqueName: \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\") pod \"c869c3e1-4241-46fc-bb31-e490e72d7805\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") "
Nov 03 19:18:28.405619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.405599    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\") pod \"c869c3e1-4241-46fc-bb31-e490e72d7805\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") "
Nov 03 19:18:28.405737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.405714    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\") pod \"c869c3e1-4241-46fc-bb31-e490e72d7805\" (UID: \"c869c3e1-4241-46fc-bb31-e490e72d7805\") "
Nov 03 19:18:28.406021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.405994    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca" pod "c869c3e1-4241-46fc-bb31-e490e72d7805" (UID: "c869c3e1-4241-46fc-bb31-e490e72d7805"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:28.406497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.406469    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:28.408236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.408207    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn" pod "c869c3e1-4241-46fc-bb31-e490e72d7805" (UID: "c869c3e1-4241-46fc-bb31-e490e72d7805"). InnerVolumeSpecName "kube-api-access-qjcmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:28.412199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.412171    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca" pod "c869c3e1-4241-46fc-bb31-e490e72d7805" (UID: "c869c3e1-4241-46fc-bb31-e490e72d7805"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:28.507470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.507439    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c869c3e1-4241-46fc-bb31-e490e72d7805-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:28.507470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.507470    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjcmn\" (UniqueName: \"kubernetes.io/projected/c869c3e1-4241-46fc-bb31-e490e72d7805-kube-api-access-qjcmn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:28.569507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.569467    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:28.918489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:28.918398    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:29.166622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.166588    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:29.188488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.188380    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"
Nov 03 19:18:29.188488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.188395    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx" event={"ID":"c869c3e1-4241-46fc-bb31-e490e72d7805","Type":"ContainerDied","Data":"941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8"}
Nov 03 19:18:29.188488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.188448    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941adcd59974eea5d0ed0794a1b995c240bfd340a98ce5e0459ba185367e47f8"
Nov 03 19:18:29.236436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.236401    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:18:29.237277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.237254    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c869c3e1-4241-46fc-bb31-e490e72d7805" containerName="revoke-keys"
Nov 03 19:18:29.237277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.237276    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c869c3e1-4241-46fc-bb31-e490e72d7805" containerName="revoke-keys"
Nov 03 19:18:29.237463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.237428    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c869c3e1-4241-46fc-bb31-e490e72d7805" containerName="revoke-keys"
Nov 03 19:18:29.241148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.241127    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.289960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.289927    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1410ed\""
Nov 03 19:18:29.290208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.290175    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1410ed-dockercfg-6hfp4\""
Nov 03 19:18:29.295393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.295367    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:18:29.327512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.327472    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc4wd\" (UniqueName: \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.328421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.327702    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.434813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.431806    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc4wd\" (UniqueName: \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.434813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.432027    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.434813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.433473    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.491737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.491704    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc4wd\" (UniqueName: \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\") pod \"payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.567527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.567494    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:29.942646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:29.942567    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:18:29.947335 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:18:29.947304    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deecb35_f8be_4fbb_ba3e_68968fd1ff79.slice/crio-3234fe021e9da1bb23a08028c85dd26f3aeea9f20969e2dc151c5d549e431d2b WatchSource:0}: Error finding container 3234fe021e9da1bb23a08028c85dd26f3aeea9f20969e2dc151c5d549e431d2b: Status 404 returned error can't find the container with id 3234fe021e9da1bb23a08028c85dd26f3aeea9f20969e2dc151c5d549e431d2b
Nov 03 19:18:30.198276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:30.198181    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" event={"ID":"9deecb35-f8be-4fbb-ba3e-68968fd1ff79","Type":"ContainerStarted","Data":"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"}
Nov 03 19:18:30.198276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:30.198247    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:30.198276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:30.198273    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" event={"ID":"9deecb35-f8be-4fbb-ba3e-68968fd1ff79","Type":"ContainerStarted","Data":"3234fe021e9da1bb23a08028c85dd26f3aeea9f20969e2dc151c5d549e431d2b"}
Nov 03 19:18:30.309093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:30.309038    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" podStartSLOduration=1.309024838 podStartE2EDuration="1.309024838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:30.305896782 +0000 UTC m=+2008.583250505" watchObservedRunningTime="2026-11-03 19:18:30.309024838 +0000 UTC m=+2008.586378420"
Nov 03 19:18:32.176518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:32.176480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:35.225464 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:18:35.225430    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:18:36.087933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.087899    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"]
Nov 03 19:18:36.092778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.092759    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.145614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.145586    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"]
Nov 03 19:18:36.186861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.186824    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:18:36.186861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.186856    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:18:36.187199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.187163    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:18:36.187413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.187386    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:18:36.228235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.228213    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2sn8\" (UniqueName: \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.228574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.228247    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.228574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.228381    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.330034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.330007    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2sn8\" (UniqueName: \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.330195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.330043    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.330195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.330119    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.330843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.330806    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.332423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.332404    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.386835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.386763    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2sn8\" (UniqueName: \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\") pod \"maas-api-revoke-sub-bc9d86a4-qpbss\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.407191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.407165    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:36.666879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.666803    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:36.782329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.782295    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"]
Nov 03 19:18:36.787127 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:18:36.787096    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148d862c_34fc_4633_9ea2_d5d690b36bd9.slice/crio-7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15 WatchSource:0}: Error finding container 7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15: Status 404 returned error can't find the container with id 7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15
Nov 03 19:18:36.920806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:36.920711    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:37.020342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.020303    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:18:37.121738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.121707    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:18:37.248662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.248621    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss" event={"ID":"148d862c-34fc-4633-9ea2-d5d690b36bd9","Type":"ContainerStarted","Data":"1416f3deacd1c01894fa7e2dbc8fcf616b8919e18578ecfb68fc4ee3ae5d90de"}
Nov 03 19:18:37.249095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.248690    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss" event={"ID":"148d862c-34fc-4633-9ea2-d5d690b36bd9","Type":"ContainerStarted","Data":"7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15"}
Nov 03 19:18:37.357972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.357919    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss" podStartSLOduration=1.357901634 podStartE2EDuration="1.357901634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:37.357148628 +0000 UTC m=+2015.634502214" watchObservedRunningTime="2026-11-03 19:18:37.357901634 +0000 UTC m=+2015.635255217"
Nov 03 19:18:37.475737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.475645    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:37.726869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:37.726778    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:38.259049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:38.259011    4944 generic.go:356] "Generic (PLEG): container finished" podID="148d862c-34fc-4633-9ea2-d5d690b36bd9" containerID="1416f3deacd1c01894fa7e2dbc8fcf616b8919e18578ecfb68fc4ee3ae5d90de" exitCode=0
Nov 03 19:18:38.259510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:38.259106    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss" event={"ID":"148d862c-34fc-4633-9ea2-d5d690b36bd9","Type":"ContainerDied","Data":"1416f3deacd1c01894fa7e2dbc8fcf616b8919e18578ecfb68fc4ee3ae5d90de"}
Nov 03 19:18:39.421050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.421023    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:39.479776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.479743    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\") pod \"148d862c-34fc-4633-9ea2-d5d690b36bd9\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") "
Nov 03 19:18:39.479977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.479886    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\" (UniqueName: \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\") pod \"148d862c-34fc-4633-9ea2-d5d690b36bd9\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") "
Nov 03 19:18:39.479977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.479960    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\") pod \"148d862c-34fc-4633-9ea2-d5d690b36bd9\" (UID: \"148d862c-34fc-4633-9ea2-d5d690b36bd9\") "
Nov 03 19:18:39.480105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.480080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca" pod "148d862c-34fc-4633-9ea2-d5d690b36bd9" (UID: "148d862c-34fc-4633-9ea2-d5d690b36bd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:39.480604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.480585    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:39.482085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.482066    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca" pod "148d862c-34fc-4633-9ea2-d5d690b36bd9" (UID: "148d862c-34fc-4633-9ea2-d5d690b36bd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:39.482168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.482085    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8" pod "148d862c-34fc-4633-9ea2-d5d690b36bd9" (UID: "148d862c-34fc-4633-9ea2-d5d690b36bd9"). InnerVolumeSpecName "kube-api-access-w2sn8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:39.582672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.582593    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148d862c-34fc-4633-9ea2-d5d690b36bd9-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:39.582672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:39.582622    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2sn8\" (UniqueName: \"kubernetes.io/projected/148d862c-34fc-4633-9ea2-d5d690b36bd9-kube-api-access-w2sn8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:40.275578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:40.275549    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"
Nov 03 19:18:40.275747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:40.275542    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss" event={"ID":"148d862c-34fc-4633-9ea2-d5d690b36bd9","Type":"ContainerDied","Data":"7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15"}
Nov 03 19:18:40.275747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:40.275671    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7185595a530359e61e040d9de7b39f6175ab6a33f49f4b0174ac68a15c272a15"
Nov 03 19:18:41.208317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:41.208243    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:18:41.603750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:41.603711    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:42.021612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.021562    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" podUID="373578f5-bb60-4487-8f65-22f9292a2a79" containerName="payload-processing" containerID="cri-o://d9bc39daa315eb5a348761e37d5290933276312fd1dffb54c827b599b82991eb" gracePeriod=25
Nov 03 19:18:42.122925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.122884    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" podUID="53a3107f-9d54-470c-8587-f4bdef15f9bc" containerName="payload-pre-processing" containerID="cri-o://bd9aa025445ab7a64716cac1d632c6e43028a6650627833c3a2bc4258f7a7e0c" gracePeriod=25
Nov 03 19:18:42.297868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.297742    4944 generic.go:356] "Generic (PLEG): container finished" podID="53a3107f-9d54-470c-8587-f4bdef15f9bc" containerID="bd9aa025445ab7a64716cac1d632c6e43028a6650627833c3a2bc4258f7a7e0c" exitCode=0
Nov 03 19:18:42.297868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.297839    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" event={"ID":"53a3107f-9d54-470c-8587-f4bdef15f9bc","Type":"ContainerDied","Data":"bd9aa025445ab7a64716cac1d632c6e43028a6650627833c3a2bc4258f7a7e0c"}
Nov 03 19:18:42.304258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.304033    4944 generic.go:356] "Generic (PLEG): container finished" podID="373578f5-bb60-4487-8f65-22f9292a2a79" containerID="d9bc39daa315eb5a348761e37d5290933276312fd1dffb54c827b599b82991eb" exitCode=0
Nov 03 19:18:42.304258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.304068    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" event={"ID":"373578f5-bb60-4487-8f65-22f9292a2a79","Type":"ContainerDied","Data":"d9bc39daa315eb5a348761e37d5290933276312fd1dffb54c827b599b82991eb"}
Nov 03 19:18:42.322967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.322945    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:18:42.418381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.418355    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:18:42.438209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.438179    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\") pod \"373578f5-bb60-4487-8f65-22f9292a2a79\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") "
Nov 03 19:18:42.438349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.438224    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\" (UniqueName: \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\") pod \"373578f5-bb60-4487-8f65-22f9292a2a79\" (UID: \"373578f5-bb60-4487-8f65-22f9292a2a79\") "
Nov 03 19:18:42.439426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.439400    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume" pod "373578f5-bb60-4487-8f65-22f9292a2a79" (UID: "373578f5-bb60-4487-8f65-22f9292a2a79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:42.440960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.440931    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms" pod "373578f5-bb60-4487-8f65-22f9292a2a79" (UID: "373578f5-bb60-4487-8f65-22f9292a2a79"). InnerVolumeSpecName "kube-api-access-26lms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:42.539462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.539429    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\") pod \"53a3107f-9d54-470c-8587-f4bdef15f9bc\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") "
Nov 03 19:18:42.539620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.539603    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\" (UniqueName: \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\") pod \"53a3107f-9d54-470c-8587-f4bdef15f9bc\" (UID: \"53a3107f-9d54-470c-8587-f4bdef15f9bc\") "
Nov 03 19:18:42.539839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.539785    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume" pod "53a3107f-9d54-470c-8587-f4bdef15f9bc" (UID: "53a3107f-9d54-470c-8587-f4bdef15f9bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:42.540196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.540177    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53a3107f-9d54-470c-8587-f4bdef15f9bc-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:42.540238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.540201    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373578f5-bb60-4487-8f65-22f9292a2a79-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:42.540238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.540211    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lms\" (UniqueName: \"kubernetes.io/projected/373578f5-bb60-4487-8f65-22f9292a2a79-kube-api-access-26lms\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:42.541720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.541697    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7" pod "53a3107f-9d54-470c-8587-f4bdef15f9bc" (UID: "53a3107f-9d54-470c-8587-f4bdef15f9bc"). InnerVolumeSpecName "kube-api-access-tjhb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:42.641462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:42.641377    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjhb7\" (UniqueName: \"kubernetes.io/projected/53a3107f-9d54-470c-8587-f4bdef15f9bc-kube-api-access-tjhb7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:43.312725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.312634    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"
Nov 03 19:18:43.312725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.312633    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8" event={"ID":"53a3107f-9d54-470c-8587-f4bdef15f9bc","Type":"ContainerDied","Data":"26a9f65a5a1d36861ea9690b0d6b0cfd62facec1e474bed286df989eb8060d63"}
Nov 03 19:18:43.313269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.312779    4944 scope.go:122] "RemoveContainer" containerID="bd9aa025445ab7a64716cac1d632c6e43028a6650627833c3a2bc4258f7a7e0c"
Nov 03 19:18:43.314214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.314191    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd" event={"ID":"373578f5-bb60-4487-8f65-22f9292a2a79","Type":"ContainerDied","Data":"7285f3e8df80fb0a6b67f8ad922811bf38d61d54af43bc1ea60180e08f6e0029"}
Nov 03 19:18:43.314282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.314236    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"
Nov 03 19:18:43.323916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.323898    4944 scope.go:122] "RemoveContainer" containerID="d9bc39daa315eb5a348761e37d5290933276312fd1dffb54c827b599b82991eb"
Nov 03 19:18:43.467254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.467216    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:18:43.514084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.514048    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4702dd-bf89cc5c9-ccwj8"]
Nov 03 19:18:43.572768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.572668    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:18:43.576297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.576270    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4702dd-9c7975c9d-8xwpd"]
Nov 03 19:18:43.936079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.935999    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373578f5-bb60-4487-8f65-22f9292a2a79" path="/var/lib/kubelet/pods/373578f5-bb60-4487-8f65-22f9292a2a79/volumes"
Nov 03 19:18:43.936716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:43.936688    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a3107f-9d54-470c-8587-f4bdef15f9bc" path="/var/lib/kubelet/pods/53a3107f-9d54-470c-8587-f4bdef15f9bc/volumes"
Nov 03 19:18:45.190220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:45.190186    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:46.724931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.724900    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:46.891305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.891272    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"]
Nov 03 19:18:46.891989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.891974    4944 cpu_manager.go:414] "containerMap: removing container" podUID="53a3107f-9d54-470c-8587-f4bdef15f9bc" containerName="payload-pre-processing"
Nov 03 19:18:46.892059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.891990    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a3107f-9d54-470c-8587-f4bdef15f9bc" containerName="payload-pre-processing"
Nov 03 19:18:46.892059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892045    4944 cpu_manager.go:414] "containerMap: removing container" podUID="148d862c-34fc-4633-9ea2-d5d690b36bd9" containerName="revoke-keys"
Nov 03 19:18:46.892059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892050    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148d862c-34fc-4633-9ea2-d5d690b36bd9" containerName="revoke-keys"
Nov 03 19:18:46.892059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892058    4944 cpu_manager.go:414] "containerMap: removing container" podUID="373578f5-bb60-4487-8f65-22f9292a2a79" containerName="payload-processing"
Nov 03 19:18:46.892189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892063    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373578f5-bb60-4487-8f65-22f9292a2a79" containerName="payload-processing"
Nov 03 19:18:46.892189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892171    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="148d862c-34fc-4633-9ea2-d5d690b36bd9" containerName="revoke-keys"
Nov 03 19:18:46.892189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892179    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="373578f5-bb60-4487-8f65-22f9292a2a79" containerName="payload-processing"
Nov 03 19:18:46.892189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.892186    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a3107f-9d54-470c-8587-f4bdef15f9bc" containerName="payload-pre-processing"
Nov 03 19:18:46.897316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.897297    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:46.950231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.950195    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"]
Nov 03 19:18:46.990952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.990920    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:18:46.991194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.990953    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:18:46.991194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.990996    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:18:46.992157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:46.992134    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:18:47.006522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.006488    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mz9c\" (UniqueName: \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.006739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.006716    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.006906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.006885    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.107773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.107731    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.107967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.107857    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mz9c\" (UniqueName: \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.108072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.108052    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.108929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.108896    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.110455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.110429    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.165248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.165207    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mz9c\" (UniqueName: \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\") pod \"maas-api-revoke-keys-e2e-api-a-08baeb-54vjm\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.211931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.211899    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:47.589129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:47.589069    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"]
Nov 03 19:18:47.592358 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:18:47.592329    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022200f3_69d6_4084_96ca_31f3cdd59294.slice/crio-15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba WatchSource:0}: Error finding container 15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba: Status 404 returned error can't find the container with id 15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba
Nov 03 19:18:48.357157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:48.357056    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm" event={"ID":"022200f3-69d6-4084-96ca-31f3cdd59294","Type":"ContainerStarted","Data":"52cf282ab4b4caabffd7be53a8e10138654b1055e7aba9fd8dd8e351a1310543"}
Nov 03 19:18:48.357157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:48.357119    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm" event={"ID":"022200f3-69d6-4084-96ca-31f3cdd59294","Type":"ContainerStarted","Data":"15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba"}
Nov 03 19:18:48.398407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:48.398371    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:18:48.398616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:48.398582    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy" containerID="cri-o://a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7" gracePeriod=30
Nov 03 19:18:48.505729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:48.505669    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm" podStartSLOduration=2.5056542 podStartE2EDuration="2.5056542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:18:48.500454105 +0000 UTC m=+2026.777807690" watchObservedRunningTime="2026-11-03 19:18:48.5056542 +0000 UTC m=+2026.783007774"
Nov 03 19:18:49.366499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:49.366465    4944 generic.go:356] "Generic (PLEG): container finished" podID="022200f3-69d6-4084-96ca-31f3cdd59294" containerID="52cf282ab4b4caabffd7be53a8e10138654b1055e7aba9fd8dd8e351a1310543" exitCode=0
Nov 03 19:18:49.366901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:49.366551    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm" event={"ID":"022200f3-69d6-4084-96ca-31f3cdd59294","Type":"ContainerDied","Data":"52cf282ab4b4caabffd7be53a8e10138654b1055e7aba9fd8dd8e351a1310543"}
Nov 03 19:18:50.531887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.531860    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:50.666893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.666812    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\") pod \"022200f3-69d6-4084-96ca-31f3cdd59294\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") "
Nov 03 19:18:50.667064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.666905    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\") pod \"022200f3-69d6-4084-96ca-31f3cdd59294\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") "
Nov 03 19:18:50.667064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.667006    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\" (UniqueName: \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\") pod \"022200f3-69d6-4084-96ca-31f3cdd59294\" (UID: \"022200f3-69d6-4084-96ca-31f3cdd59294\") "
Nov 03 19:18:50.667191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.667138    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca" pod "022200f3-69d6-4084-96ca-31f3cdd59294" (UID: "022200f3-69d6-4084-96ca-31f3cdd59294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:50.667774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.667753    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:50.669152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.669129    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c" pod "022200f3-69d6-4084-96ca-31f3cdd59294" (UID: "022200f3-69d6-4084-96ca-31f3cdd59294"). InnerVolumeSpecName "kube-api-access-2mz9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:50.669520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.669496    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca" pod "022200f3-69d6-4084-96ca-31f3cdd59294" (UID: "022200f3-69d6-4084-96ca-31f3cdd59294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:18:50.769614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.769567    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/022200f3-69d6-4084-96ca-31f3cdd59294-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:50.769614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:50.769612    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mz9c\" (UniqueName: \"kubernetes.io/projected/022200f3-69d6-4084-96ca-31f3cdd59294-kube-api-access-2mz9c\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:51.385568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:51.385535    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"
Nov 03 19:18:51.385754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:51.385561    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm" event={"ID":"022200f3-69d6-4084-96ca-31f3cdd59294","Type":"ContainerDied","Data":"15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba"}
Nov 03 19:18:51.385754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:51.385615    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15752a5dc571fdf7760263ec9521d58bc2bfe611a6bbb0b409083bb76d92daba"
Nov 03 19:18:51.756624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:51.756586    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:53.672167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.672102    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:18:53.811521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811484    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811594    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811628    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811690    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811722    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811742    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:53.811923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811839    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811874    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.811923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811886    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:53.812171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.811927    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.812171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812010    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\") pod \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\" (UID: \"3d993953-d8bc-4c7d-bee4-fd269048ce7f\") "
Nov 03 19:18:53.812171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812051    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:53.812171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812065    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:53.812406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812381    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:18:53.812759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812742    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.812860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812763    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.812860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812777    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.812860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812803    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.812860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.812817    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.813841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.813817    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:18:53.814157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.814137    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "kube-api-access-dcqv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:53.814280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.814259    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:18:53.814331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.814280    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token" pod "3d993953-d8bc-4c7d-bee4-fd269048ce7f" (UID: "3d993953-d8bc-4c7d-bee4-fd269048ce7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:18:53.914451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.914414    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.914451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.914448    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.914451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.914457    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:53.914696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:53.914466    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqv6\" (UniqueName: \"kubernetes.io/projected/3d993953-d8bc-4c7d-bee4-fd269048ce7f-kube-api-access-dcqv6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:18:54.412158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.412125    4944 generic.go:356] "Generic (PLEG): container finished" podID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerID="a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7" exitCode=0
Nov 03 19:18:54.412335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.412173    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"
Nov 03 19:18:54.412335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.412214    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" event={"ID":"3d993953-d8bc-4c7d-bee4-fd269048ce7f","Type":"ContainerDied","Data":"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"}
Nov 03 19:18:54.412335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.412272    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x" event={"ID":"3d993953-d8bc-4c7d-bee4-fd269048ce7f","Type":"ContainerDied","Data":"cd639cde269e37bb7e1d21546265407dd07228720bd5fdd865c182cefcf9a93d"}
Nov 03 19:18:54.412335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.412296    4944 scope.go:122] "RemoveContainer" containerID="a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"
Nov 03 19:18:54.422619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.422601    4944 scope.go:122] "RemoveContainer" containerID="a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"
Nov 03 19:18:54.422920 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:18:54.422901    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7\": container with ID starting with a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7 not found: ID does not exist" containerID="a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"
Nov 03 19:18:54.422996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.422929    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7"} err="failed to get container status \"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7\": rpc error: code = NotFound desc = could not find container \"a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7\": container with ID starting with a88af3a601f03250cc269c4c83944a0eb51d6a376542eb1bac4e06ce930913b7 not found: ID does not exist"
Nov 03 19:18:54.615380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.615330    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:18:54.659949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:54.659920    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-scr2x"]
Nov 03 19:18:55.947254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:55.947219    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" path="/var/lib/kubelet/pods/3d993953-d8bc-4c7d-bee4-fd269048ce7f/volumes"
Nov 03 19:18:56.684488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:56.684455    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:58.490629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:58.490593    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:18:58.591621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:58.591583    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:18:58.945035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:58.944955    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:18:59.190031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:18:59.190001    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:01.797070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:01.797034    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:03.491823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.491765    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" podUID="d829026c-b5c2-427e-b43a-278701572fba" containerName="payload-processing" containerID="cri-o://382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289" gracePeriod=25
Nov 03 19:19:03.592344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.592289    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" podUID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" containerName="payload-pre-processing" containerID="cri-o://96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914" gracePeriod=25
Nov 03 19:19:03.602076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.602048    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:19:03.602284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.602261    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy" containerID="cri-o://00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73" gracePeriod=30
Nov 03 19:19:03.795540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.795513    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:19:03.871372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.871310    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\") pod \"d829026c-b5c2-427e-b43a-278701572fba\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") "
Nov 03 19:19:03.871372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.871367    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\" (UniqueName: \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\") pod \"d829026c-b5c2-427e-b43a-278701572fba\" (UID: \"d829026c-b5c2-427e-b43a-278701572fba\") "
Nov 03 19:19:03.871844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.871780    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume" pod "d829026c-b5c2-427e-b43a-278701572fba" (UID: "d829026c-b5c2-427e-b43a-278701572fba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:03.872758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.872732    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d829026c-b5c2-427e-b43a-278701572fba-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:03.874061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.874033    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5" pod "d829026c-b5c2-427e-b43a-278701572fba" (UID: "d829026c-b5c2-427e-b43a-278701572fba"). InnerVolumeSpecName "kube-api-access-2f9z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:03.884135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.884114    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:19:03.973614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.973583    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\") pod \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") "
Nov 03 19:19:03.973784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.973710    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\" (UniqueName: \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\") pod \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\" (UID: \"15f1e3c9-b2d0-4e70-a62e-2da984c783e5\") "
Nov 03 19:19:03.973971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.973948    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume" pod "15f1e3c9-b2d0-4e70-a62e-2da984c783e5" (UID: "15f1e3c9-b2d0-4e70-a62e-2da984c783e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:03.974241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.974220    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:03.974300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.974247    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9z5\" (UniqueName: \"kubernetes.io/projected/d829026c-b5c2-427e-b43a-278701572fba-kube-api-access-2f9z5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:03.975769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:03.975751    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6" pod "15f1e3c9-b2d0-4e70-a62e-2da984c783e5" (UID: "15f1e3c9-b2d0-4e70-a62e-2da984c783e5"). InnerVolumeSpecName "kube-api-access-ll4v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:04.075589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.075490    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll4v6\" (UniqueName: \"kubernetes.io/projected/15f1e3c9-b2d0-4e70-a62e-2da984c783e5-kube-api-access-ll4v6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:04.250842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.250812    4944 scope.go:122] "RemoveContainer" containerID="8cc6a2964453edce66eefc8cf00c5e91520d321eb26a2d483f3697c7d8e4169e"
Nov 03 19:19:04.509243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.509209    4944 generic.go:356] "Generic (PLEG): container finished" podID="d829026c-b5c2-427e-b43a-278701572fba" containerID="382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289" exitCode=0
Nov 03 19:19:04.509658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.509258    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"
Nov 03 19:19:04.509658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.509295    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" event={"ID":"d829026c-b5c2-427e-b43a-278701572fba","Type":"ContainerDied","Data":"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"}
Nov 03 19:19:04.509658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.509353    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7" event={"ID":"d829026c-b5c2-427e-b43a-278701572fba","Type":"ContainerDied","Data":"5320591462e5928620a4a55f5a26bb9282dac7fcb8b055997d7cb2266da9e70b"}
Nov 03 19:19:04.509658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.509380    4944 scope.go:122] "RemoveContainer" containerID="382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"
Nov 03 19:19:04.511075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.511039    4944 generic.go:356] "Generic (PLEG): container finished" podID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" containerID="96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914" exitCode=0
Nov 03 19:19:04.511206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.511103    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"
Nov 03 19:19:04.511206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.511131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" event={"ID":"15f1e3c9-b2d0-4e70-a62e-2da984c783e5","Type":"ContainerDied","Data":"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"}
Nov 03 19:19:04.511206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.511165    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm" event={"ID":"15f1e3c9-b2d0-4e70-a62e-2da984c783e5","Type":"ContainerDied","Data":"dfe5699057b3a0a8d16c7fe7e257af16a2d3fd9c9ccfe3e548015bdc65081fdc"}
Nov 03 19:19:04.522176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.522141    4944 scope.go:122] "RemoveContainer" containerID="382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"
Nov 03 19:19:04.522518 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:04.522492    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289\": container with ID starting with 382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289 not found: ID does not exist" containerID="382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"
Nov 03 19:19:04.522634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.522525    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289"} err="failed to get container status \"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289\": rpc error: code = NotFound desc = could not find container \"382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289\": container with ID starting with 382b3f66b216168dbaef06ccb8363d4e62140dbb2ce68fdee58dc4125a341289 not found: ID does not exist"
Nov 03 19:19:04.522634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.522549    4944 scope.go:122] "RemoveContainer" containerID="96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"
Nov 03 19:19:04.533044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.533026    4944 scope.go:122] "RemoveContainer" containerID="96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"
Nov 03 19:19:04.533349 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:04.533330    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914\": container with ID starting with 96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914 not found: ID does not exist" containerID="96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"
Nov 03 19:19:04.533414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.533361    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914"} err="failed to get container status \"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914\": rpc error: code = NotFound desc = could not find container \"96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914\": container with ID starting with 96370571677d8d0ce8af6faaf1e9cd0086d636fe08b0eaedb6c3bf59a3e19914 not found: ID does not exist"
Nov 03 19:19:04.664999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.664968    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:19:04.668480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.668458    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-08baeb-7d4db6655-2g9wm"]
Nov 03 19:19:04.770780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.770702    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:19:04.774360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:04.774333    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08baeb-b44658846-k5qm7"]
Nov 03 19:19:05.227953 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:05.227866    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:19:05.935938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:05.935907    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" path="/var/lib/kubelet/pods/15f1e3c9-b2d0-4e70-a62e-2da984c783e5/volumes"
Nov 03 19:19:05.936419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:05.936403    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d829026c-b5c2-427e-b43a-278701572fba" path="/var/lib/kubelet/pods/d829026c-b5c2-427e-b43a-278701572fba/volumes"
Nov 03 19:19:06.549217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:06.549171    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:08.872996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:08.872939    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:19:09.044109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044078    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044174    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044219    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044299    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044325    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044456    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044491    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044486    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:19:09.044592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044528    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044580    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\") pod \"5b35487a-7313-4827-9384-563cc1670fd7\" (UID: \"5b35487a-7313-4827-9384-563cc1670fd7\") "
Nov 03 19:19:09.044825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044703    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044921    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.044939    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045414    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045451    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045466    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045479    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b35487a-7313-4827-9384-563cc1670fd7-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.045770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.045492    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.046958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.046929    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:19:09.046958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.046942    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:09.047535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.047510    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:19:09.048172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.048146    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r" pod "5b35487a-7313-4827-9384-563cc1670fd7" (UID: "5b35487a-7313-4827-9384-563cc1670fd7"). InnerVolumeSpecName "kube-api-access-9nc5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:09.147341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.147272    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nc5r\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-kube-api-access-9nc5r\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.147341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.147300    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b35487a-7313-4827-9384-563cc1670fd7-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.147341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.147309    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b35487a-7313-4827-9384-563cc1670fd7-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.147341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.147317    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b35487a-7313-4827-9384-563cc1670fd7-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:09.551137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.551098    4944 generic.go:356] "Generic (PLEG): container finished" podID="5b35487a-7313-4827-9384-563cc1670fd7" containerID="00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73" exitCode=0
Nov 03 19:19:09.551329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.551166    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"
Nov 03 19:19:09.551329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.551206    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" event={"ID":"5b35487a-7313-4827-9384-563cc1670fd7","Type":"ContainerDied","Data":"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"}
Nov 03 19:19:09.551329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.551272    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4" event={"ID":"5b35487a-7313-4827-9384-563cc1670fd7","Type":"ContainerDied","Data":"1cba3efe3ba1c64f02d4aa22fb5858c3a72c572e8feb41fdab2c852a79e8296a"}
Nov 03 19:19:09.551329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.551291    4944 scope.go:122] "RemoveContainer" containerID="00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"
Nov 03 19:19:09.563172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.563150    4944 scope.go:122] "RemoveContainer" containerID="00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"
Nov 03 19:19:09.563435 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:09.563418    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73\": container with ID starting with 00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73 not found: ID does not exist" containerID="00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"
Nov 03 19:19:09.563498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.563443    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73"} err="failed to get container status \"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73\": rpc error: code = NotFound desc = could not find container \"00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73\": container with ID starting with 00630e7ab1a98ddcc429d81dabbba9f099508b8fca0e966726499d3111be6e73 not found: ID does not exist"
Nov 03 19:19:09.707820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.707773    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:19:09.754097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.754060    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-78zb4"]
Nov 03 19:19:09.936393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:09.936316    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b35487a-7313-4827-9384-563cc1670fd7" path="/var/lib/kubelet/pods/5b35487a-7313-4827-9384-563cc1670fd7/volumes"
Nov 03 19:19:11.033917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:11.033882    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:14.994997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:14.994962    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:16.915751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:16.915713    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:17.178544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.178449    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:17.439194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.439107    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:17.459689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.459649    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"]
Nov 03 19:19:17.503920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.503888    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4702dd-hsvpx"]
Nov 03 19:19:17.630721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.630687    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"]
Nov 03 19:19:17.631498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631478    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy"
Nov 03 19:19:17.631572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631500    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy"
Nov 03 19:19:17.631572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631539    4944 cpu_manager.go:414] "containerMap: removing container" podUID="022200f3-69d6-4084-96ca-31f3cdd59294" containerName="revoke-keys"
Nov 03 19:19:17.631572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631546    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022200f3-69d6-4084-96ca-31f3cdd59294" containerName="revoke-keys"
Nov 03 19:19:17.631572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631561    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy"
Nov 03 19:19:17.631572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631567    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631574    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d829026c-b5c2-427e-b43a-278701572fba" containerName="payload-processing"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631580    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d829026c-b5c2-427e-b43a-278701572fba" containerName="payload-processing"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631597    4944 cpu_manager.go:414] "containerMap: removing container" podUID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" containerName="payload-pre-processing"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631601    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" containerName="payload-pre-processing"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631694    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b35487a-7313-4827-9384-563cc1670fd7" containerName="istio-proxy"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631708    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="022200f3-69d6-4084-96ca-31f3cdd59294" containerName="revoke-keys"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631718    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d993953-d8bc-4c7d-bee4-fd269048ce7f" containerName="istio-proxy"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631724    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d829026c-b5c2-427e-b43a-278701572fba" containerName="payload-processing"
Nov 03 19:19:17.631745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.631732    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="15f1e3c9-b2d0-4e70-a62e-2da984c783e5" containerName="payload-pre-processing"
Nov 03 19:19:17.636888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.636865    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.685604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.685573    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:19:17.685998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.685965    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:19:17.686190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.685969    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:19:17.686334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.686012    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:19:17.689252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.689187    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"]
Nov 03 19:19:17.690541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.690515    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.690654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.690640    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9tg\" (UniqueName: \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.690722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.690709    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.791682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.791651    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.791882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.791754    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9tg\" (UniqueName: \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.791882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.791849    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.792465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.792436    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.794210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.794185    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.848886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.848853    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9tg\" (UniqueName: \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\") pod \"maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.937866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.937834    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c869c3e1-4241-46fc-bb31-e490e72d7805" path="/var/lib/kubelet/pods/c869c3e1-4241-46fc-bb31-e490e72d7805/volumes"
Nov 03 19:19:17.952940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.952872    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:17.969278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:17.969249    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:18.327850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:18.327799    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"]
Nov 03 19:19:18.332988 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:19:18.332952    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa87235_7238_404e_b5e1_18c9162f32ff.slice/crio-9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e WatchSource:0}: Error finding container 9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e: Status 404 returned error can't find the container with id 9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e
Nov 03 19:19:18.618099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:18.618068    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx" event={"ID":"2aa87235-7238-404e-b5e1-18c9162f32ff","Type":"ContainerStarted","Data":"9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e"}
Nov 03 19:19:19.627458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:19.627419    4944 generic.go:356] "Generic (PLEG): container finished" podID="2aa87235-7238-404e-b5e1-18c9162f32ff" containerID="f909777d825ef5704b006b977bd88af471bccb51cd9b90e61b5667047e45ca8c" exitCode=0
Nov 03 19:19:19.628140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:19.627498    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx" event={"ID":"2aa87235-7238-404e-b5e1-18c9162f32ff","Type":"ContainerDied","Data":"f909777d825ef5704b006b977bd88af471bccb51cd9b90e61b5667047e45ca8c"}
Nov 03 19:19:20.785182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.785154    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:20.936552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.936464    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\") pod \"2aa87235-7238-404e-b5e1-18c9162f32ff\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") "
Nov 03 19:19:20.936737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.936663    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\") pod \"2aa87235-7238-404e-b5e1-18c9162f32ff\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") "
Nov 03 19:19:20.936737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.936701    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\" (UniqueName: \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\") pod \"2aa87235-7238-404e-b5e1-18c9162f32ff\" (UID: \"2aa87235-7238-404e-b5e1-18c9162f32ff\") "
Nov 03 19:19:20.937091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.937065    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca" pod "2aa87235-7238-404e-b5e1-18c9162f32ff" (UID: "2aa87235-7238-404e-b5e1-18c9162f32ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:20.937340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.937316    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:20.938700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.938676    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca" pod "2aa87235-7238-404e-b5e1-18c9162f32ff" (UID: "2aa87235-7238-404e-b5e1-18c9162f32ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:20.939051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:20.939021    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg" pod "2aa87235-7238-404e-b5e1-18c9162f32ff" (UID: "2aa87235-7238-404e-b5e1-18c9162f32ff"). InnerVolumeSpecName "kube-api-access-6h9tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:21.039095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.039062    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aa87235-7238-404e-b5e1-18c9162f32ff-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:21.039095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.039096    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9tg\" (UniqueName: \"kubernetes.io/projected/2aa87235-7238-404e-b5e1-18c9162f32ff-kube-api-access-6h9tg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:21.642843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.642783    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx" event={"ID":"2aa87235-7238-404e-b5e1-18c9162f32ff","Type":"ContainerDied","Data":"9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e"}
Nov 03 19:19:21.643023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.642863    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a3bdc830ef690dbc2930f41e0d6cc5935dfbeb360149e71807f7f4b7881d34e"
Nov 03 19:19:21.643023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.642808    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"
Nov 03 19:19:21.783305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:21.783268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:24.309316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:24.309281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:24.570859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:24.570718    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:24.825644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:24.825544    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:25.100238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.100160    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"]
Nov 03 19:19:25.101229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.101207    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2aa87235-7238-404e-b5e1-18c9162f32ff" containerName="revoke-keys"
Nov 03 19:19:25.101299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.101232    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa87235-7238-404e-b5e1-18c9162f32ff" containerName="revoke-keys"
Nov 03 19:19:25.101446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.101435    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa87235-7238-404e-b5e1-18c9162f32ff" containerName="revoke-keys"
Nov 03 19:19:25.105506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.105486    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.155230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.155198    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"]
Nov 03 19:19:25.197026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.196993    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:19:25.197193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.197085    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:19:25.201003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.200983    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsz2\" (UniqueName: \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.201158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.201137    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.201200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.201183    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.223633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.223608    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:19:25.223899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.223882    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:19:25.302774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.302733    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsz2\" (UniqueName: \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.302975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.302860    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.302975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.302893    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.303546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.303519    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.305438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.305416    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.359850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.359747    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsz2\" (UniqueName: \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.422100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.422066    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:25.797571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:25.797534    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"]
Nov 03 19:19:25.802845 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:19:25.802814    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd0f9ef_2c13_4963_b7c1_c1de74b1f30d.slice/crio-c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa WatchSource:0}: Error finding container c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa: Status 404 returned error can't find the container with id c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa
Nov 03 19:19:26.701049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:26.701007    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7" event={"ID":"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d","Type":"ContainerStarted","Data":"108ab1e8b3fa6d3f887a7ff69f838a52c50d0dd4d6899f9a8e253517605768d1"}
Nov 03 19:19:26.701582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:26.701128    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7" event={"ID":"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d","Type":"ContainerStarted","Data":"c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa"}
Nov 03 19:19:26.716218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:26.716179    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:26.810376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:26.810327    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7" podStartSLOduration=1.8103114009999999 podStartE2EDuration="1.810311401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:26.808636806 +0000 UTC m=+2065.085990390" watchObservedRunningTime="2026-11-03 19:19:26.810311401 +0000 UTC m=+2065.087664985"
Nov 03 19:19:27.711214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:27.711169    4944 generic.go:356] "Generic (PLEG): container finished" podID="bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" containerID="108ab1e8b3fa6d3f887a7ff69f838a52c50d0dd4d6899f9a8e253517605768d1" exitCode=0
Nov 03 19:19:27.711613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:27.711250    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7" event={"ID":"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d","Type":"ContainerDied","Data":"108ab1e8b3fa6d3f887a7ff69f838a52c50d0dd4d6899f9a8e253517605768d1"}
Nov 03 19:19:28.678492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.678458    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:28.873849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.873820    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:28.954690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.954599    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\") pod \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") "
Nov 03 19:19:28.954900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.954701    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\" (UniqueName: \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\") pod \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") "
Nov 03 19:19:28.954900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.954820    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\") pod \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\" (UID: \"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d\") "
Nov 03 19:19:28.955190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.955120    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca" pod "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" (UID: "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:28.955442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.955421    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:28.957298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.957271    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca" pod "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" (UID: "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:28.957298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:28.957283    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2" pod "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" (UID: "bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d"). InnerVolumeSpecName "kube-api-access-bfsz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:29.056610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.056568    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:29.056610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.056602    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsz2\" (UniqueName: \"kubernetes.io/projected/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d-kube-api-access-bfsz2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:29.727384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.727348    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7" event={"ID":"bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d","Type":"ContainerDied","Data":"c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa"}
Nov 03 19:19:29.727384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.727368    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"
Nov 03 19:19:29.727643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.727404    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8770ab8f59f087ad86ceb751559bef6ca9beb82db5a93e74e7a17d94d3309fa"
Nov 03 19:19:29.891924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:29.891880    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:30.138305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:30.138271    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:31.934393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:31.934358    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:35.225889 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:35.225855    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:19:35.740389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:35.740353    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:36.048644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:36.048561    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"]
Nov 03 19:19:36.128838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:36.128783    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6e520a-xcjqx"]
Nov 03 19:19:36.305243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:36.305158    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:36.557718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:36.557626    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:36.982897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:36.982816    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:37.228344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:37.228308    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:37.937297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:37.937264    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e0a356-3c66-49df-8b8c-8d5729701257" path="/var/lib/kubelet/pods/80e0a356-3c66-49df-8b8c-8d5729701257/volumes"
Nov 03 19:19:39.663371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.663336    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"]
Nov 03 19:19:39.664084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.664069    4944 cpu_manager.go:414] "containerMap: removing container" podUID="bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" containerName="revoke-keys"
Nov 03 19:19:39.664140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.664085    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" containerName="revoke-keys"
Nov 03 19:19:39.664214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.664204    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" containerName="revoke-keys"
Nov 03 19:19:39.668071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.668054    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.722738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.722704    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"]
Nov 03 19:19:39.763532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.763493    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:19:39.763532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.763519    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:19:39.763762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.763551    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:19:39.763762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.763731    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:19:39.815318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.815283    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7c8w\" (UniqueName: \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.815605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.815422    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.815605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.815575    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.917324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.917217    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7c8w\" (UniqueName: \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.917502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.917338    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.917502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.917455    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.918011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.917980    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.919833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.919808    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.974781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.974748    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7c8w\" (UniqueName: \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\") pod \"maas-api-revoke-keys-e2e-api-a-bbccf3-j87st\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:39.982326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:39.982297    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:40.371867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:40.371582    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"]
Nov 03 19:19:40.407551 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:19:40.407495    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cad21f_8b9e_4776_a66f_92ddc697acdc.slice/crio-3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29 WatchSource:0}: Error finding container 3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29: Status 404 returned error can't find the container with id 3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29
Nov 03 19:19:40.809486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:40.809444    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st" event={"ID":"a3cad21f-8b9e-4776-a66f-92ddc697acdc","Type":"ContainerStarted","Data":"fd5b7a8555a0d4ce96616812873d2836ad7b4bda41e03a2ba93f65674a48850b"}
Nov 03 19:19:40.809910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:40.809545    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st" event={"ID":"a3cad21f-8b9e-4776-a66f-92ddc697acdc","Type":"ContainerStarted","Data":"3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29"}
Nov 03 19:19:40.920653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:40.920551    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st" podStartSLOduration=1.92053613 podStartE2EDuration="1.92053613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:19:40.91775734 +0000 UTC m=+2079.195110924" watchObservedRunningTime="2026-11-03 19:19:40.92053613 +0000 UTC m=+2079.197889713"
Nov 03 19:19:41.818146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:41.818110    4944 generic.go:356] "Generic (PLEG): container finished" podID="a3cad21f-8b9e-4776-a66f-92ddc697acdc" containerID="fd5b7a8555a0d4ce96616812873d2836ad7b4bda41e03a2ba93f65674a48850b" exitCode=0
Nov 03 19:19:41.818555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:41.818155    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st" event={"ID":"a3cad21f-8b9e-4776-a66f-92ddc697acdc","Type":"ContainerDied","Data":"fd5b7a8555a0d4ce96616812873d2836ad7b4bda41e03a2ba93f65674a48850b"}
Nov 03 19:19:42.313143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:42.313108    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:42.979067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:42.979043    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:43.066618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.066580    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\") pod \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") "
Nov 03 19:19:43.066848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.066674    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\" (UniqueName: \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\") pod \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") "
Nov 03 19:19:43.066930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.066884    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\") pod \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\" (UID: \"a3cad21f-8b9e-4776-a66f-92ddc697acdc\") "
Nov 03 19:19:43.067234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.067202    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca" pod "a3cad21f-8b9e-4776-a66f-92ddc697acdc" (UID: "a3cad21f-8b9e-4776-a66f-92ddc697acdc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:43.068191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.068165    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:43.069081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.069062    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w" pod "a3cad21f-8b9e-4776-a66f-92ddc697acdc" (UID: "a3cad21f-8b9e-4776-a66f-92ddc697acdc"). InnerVolumeSpecName "kube-api-access-w7c8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:43.069249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.069229    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca" pod "a3cad21f-8b9e-4776-a66f-92ddc697acdc" (UID: "a3cad21f-8b9e-4776-a66f-92ddc697acdc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:19:43.169185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.169153    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3cad21f-8b9e-4776-a66f-92ddc697acdc-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:43.169185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.169182    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7c8w\" (UniqueName: \"kubernetes.io/projected/a3cad21f-8b9e-4776-a66f-92ddc697acdc-kube-api-access-w7c8w\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:43.843347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.843310    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"
Nov 03 19:19:43.843347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.843326    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st" event={"ID":"a3cad21f-8b9e-4776-a66f-92ddc697acdc","Type":"ContainerDied","Data":"3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29"}
Nov 03 19:19:43.843576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:43.843418    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a3db23dc9f79493953bcdf460c2ced6984c16f690895461863ce1df0164cf29"
Nov 03 19:19:46.606748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:46.606710    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:48.443661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:48.443620    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:48.737670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:48.737555    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:48.854480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:48.854438    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"]
Nov 03 19:19:48.897155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:48.897127    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4f1a90-nw9f7"]
Nov 03 19:19:48.989075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:48.988983    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:49.251205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:49.251160    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:49.504728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:49.504644    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:49.935878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:49.935782    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d" path="/var/lib/kubelet/pods/bbd0f9ef-2c13-4963-b7c1-c1de74b1f30d/volumes"
Nov 03 19:19:50.548277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:50.548241    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:19:50.806457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:50.806373    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:19:51.076205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:51.076096    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:51.321699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:51.321663    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:51.735353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:51.735300    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:51.977006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:51.976970    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:54.355202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:54.355163    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:55.548603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.548541    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" podUID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" containerName="payload-processing" containerID="cri-o://176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828" gracePeriod=25
Nov 03 19:19:55.797604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.797574    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:55.807667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.807604    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" podUID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" containerName="payload-pre-processing" containerID="cri-o://502e0a391144e0864abfbc46ed4858f4ac9d3d48a4ce5ff619dcc0f4c004c300" gracePeriod=25
Nov 03 19:19:55.818037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.818016    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:19:55.877252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.877210    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\") pod \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") "
Nov 03 19:19:55.877414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.877363    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\" (UniqueName: \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\") pod \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\" (UID: \"634d6fee-d95b-443d-8ee7-25035c2fc0ce\") "
Nov 03 19:19:55.877612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.877588    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume" pod "634d6fee-d95b-443d-8ee7-25035c2fc0ce" (UID: "634d6fee-d95b-443d-8ee7-25035c2fc0ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:55.877928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.877912    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/634d6fee-d95b-443d-8ee7-25035c2fc0ce-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:55.879482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.879461    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln" pod "634d6fee-d95b-443d-8ee7-25035c2fc0ce" (UID: "634d6fee-d95b-443d-8ee7-25035c2fc0ce"). InnerVolumeSpecName "kube-api-access-wpqln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:55.961316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.959593    4944 generic.go:356] "Generic (PLEG): container finished" podID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" containerID="176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828" exitCode=0
Nov 03 19:19:55.961316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.959686    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" event={"ID":"634d6fee-d95b-443d-8ee7-25035c2fc0ce","Type":"ContainerDied","Data":"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"}
Nov 03 19:19:55.961316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.959723    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9" event={"ID":"634d6fee-d95b-443d-8ee7-25035c2fc0ce","Type":"ContainerDied","Data":"c8467a5394318b534782e6b533ed2437ec6def65a3950961200666e2d81d74ad"}
Nov 03 19:19:55.961316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.959743    4944 scope.go:122] "RemoveContainer" containerID="176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"
Nov 03 19:19:55.961316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.959896    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"
Nov 03 19:19:55.964510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.964356    4944 generic.go:356] "Generic (PLEG): container finished" podID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" containerID="502e0a391144e0864abfbc46ed4858f4ac9d3d48a4ce5ff619dcc0f4c004c300" exitCode=0
Nov 03 19:19:55.964510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.964468    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" event={"ID":"bfcab9c7-614e-479e-8793-ba2d1d9b6427","Type":"ContainerDied","Data":"502e0a391144e0864abfbc46ed4858f4ac9d3d48a4ce5ff619dcc0f4c004c300"}
Nov 03 19:19:55.978072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.977864    4944 scope.go:122] "RemoveContainer" containerID="176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"
Nov 03 19:19:55.978291 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:19:55.978274    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828\": container with ID starting with 176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828 not found: ID does not exist" containerID="176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"
Nov 03 19:19:55.978427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.978380    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828"} err="failed to get container status \"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828\": rpc error: code = NotFound desc = could not find container \"176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828\": container with ID starting with 176a39367445cd25a96a3d207593ac061a97bb1478042056d75d95c271081828 not found: ID does not exist"
Nov 03 19:19:55.980484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:55.980461    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpqln\" (UniqueName: \"kubernetes.io/projected/634d6fee-d95b-443d-8ee7-25035c2fc0ce-kube-api-access-wpqln\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:56.062509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.062420    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:56.097635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.097610    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:19:56.131980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.131949    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:19:56.135448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.135418    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bbccf3-77579495b-q2vk9"]
Nov 03 19:19:56.209822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.207545    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\") pod \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") "
Nov 03 19:19:56.209822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.207694    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\" (UniqueName: \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\") pod \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\" (UID: \"bfcab9c7-614e-479e-8793-ba2d1d9b6427\") "
Nov 03 19:19:56.214820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.213336    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q" pod "bfcab9c7-614e-479e-8793-ba2d1d9b6427" (UID: "bfcab9c7-614e-479e-8793-ba2d1d9b6427"). InnerVolumeSpecName "kube-api-access-z6r4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:19:56.214820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.214427    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume" pod "bfcab9c7-614e-479e-8793-ba2d1d9b6427" (UID: "bfcab9c7-614e-479e-8793-ba2d1d9b6427"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:19:56.314129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.314025    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6r4q\" (UniqueName: \"kubernetes.io/projected/bfcab9c7-614e-479e-8793-ba2d1d9b6427-kube-api-access-z6r4q\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:56.314129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.314057    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfcab9c7-614e-479e-8793-ba2d1d9b6427-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:19:56.322932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.322899    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:56.973250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.973177    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"
Nov 03 19:19:56.973659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.973184    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p" event={"ID":"bfcab9c7-614e-479e-8793-ba2d1d9b6427","Type":"ContainerDied","Data":"6f576247c181027897cff5fd6f5bd5511659293b6cdedc89b0362d31c4f647ec"}
Nov 03 19:19:56.973659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:56.973302    4944 scope.go:122] "RemoveContainer" containerID="502e0a391144e0864abfbc46ed4858f4ac9d3d48a4ce5ff619dcc0f4c004c300"
Nov 03 19:19:57.064131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.064100    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:57.129713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.129673    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:19:57.135686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.135655    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bbccf3-6484b95fdc-cz25p"]
Nov 03 19:19:57.696165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.696124    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:57.935989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.935958    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" path="/var/lib/kubelet/pods/634d6fee-d95b-443d-8ee7-25035c2fc0ce/volumes"
Nov 03 19:19:57.936450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:57.936427    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" path="/var/lib/kubelet/pods/bfcab9c7-614e-479e-8793-ba2d1d9b6427/volumes"
Nov 03 19:19:58.227675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:58.227640    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:19:59.476781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:19:59.476741    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:01.294936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:01.294884    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:01.542108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:01.542072    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:02.044079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:02.044027    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:02.254883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:02.254853    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:20:02.258734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:02.258711    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:20:02.293375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:02.293342    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:04.355361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:04.355325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:05.224920 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:20:05.224884    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:20:05.469915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:05.469873    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:05.720004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:05.719961    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:08.684806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:08.684746    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:10.166568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:10.166530    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:10.410914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:10.410873    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:10.544478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:10.544441    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"]
Nov 03 19:20:10.590655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:10.590620    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-rh46h"]
Nov 03 19:20:10.664965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:10.664933    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:11.937909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:11.937876    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ed8f96-8da1-4dfa-9b07-819ee5273af6" path="/var/lib/kubelet/pods/a3ed8f96-8da1-4dfa-9b07-819ee5273af6/volumes"
Nov 03 19:20:12.837252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:12.837216    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:13.086471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:13.086438    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:14.628156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:14.628116    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:15.153290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:15.153253    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:17.546427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:17.546384    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:17.795091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:17.795056    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:18.062038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:18.062001    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:18.330144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:18.329962    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:20.214316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:20.214278    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:20.750343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:20.750289    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:20.997637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:20.997600    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:25.430333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:25.430297    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:29.026073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:29.026037    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:30.260761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:30.260719    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:35.185518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:35.183022    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:35.226199 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:20:35.226161    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:20:39.177223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:39.177168    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:40.723952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:40.723913    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:40.979842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:40.979720    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:41.120281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:41.120249    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"]
Nov 03 19:20:41.166135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:41.166106    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3d4656-7qhfx"]
Nov 03 19:20:41.226858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:41.226824    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:41.598228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:41.598196    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:41.937995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:41.937918    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa87235-7238-404e-b5e1-18c9162f32ff" path="/var/lib/kubelet/pods/2aa87235-7238-404e-b5e1-18c9162f32ff/volumes"
Nov 03 19:20:46.096650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:46.096610    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:46.339496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:46.339458    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:20:46.339700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:46.339674    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy" containerID="cri-o://89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab" gracePeriod=30
Nov 03 19:20:47.989846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:47.989815    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:48.238662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:48.238623    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:48.490686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:48.490653    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:49.304055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:49.304016    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:20:49.304505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:49.304271    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy" containerID="cri-o://7b0b0f2697c4b2b453d66e23968da2f7a14120eaa9063d40dce5a0dc0b861589" gracePeriod=30
Nov 03 19:20:49.349942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:49.349906    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:20:49.350394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:49.350363    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy" containerID="cri-o://1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2" gracePeriod=30
Nov 03 19:20:50.096551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.096512    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"]
Nov 03 19:20:50.097508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097484    4944 cpu_manager.go:414] "containerMap: removing container" podUID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" containerName="payload-processing"
Nov 03 19:20:50.097593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097510    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" containerName="payload-processing"
Nov 03 19:20:50.097593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097580    4944 cpu_manager.go:414] "containerMap: removing container" podUID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" containerName="payload-pre-processing"
Nov 03 19:20:50.097593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097589    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" containerName="payload-pre-processing"
Nov 03 19:20:50.097707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097625    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a3cad21f-8b9e-4776-a66f-92ddc697acdc" containerName="revoke-keys"
Nov 03 19:20:50.097707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097632    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cad21f-8b9e-4776-a66f-92ddc697acdc" containerName="revoke-keys"
Nov 03 19:20:50.097769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097752    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="634d6fee-d95b-443d-8ee7-25035c2fc0ce" containerName="payload-processing"
Nov 03 19:20:50.097830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097772    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcab9c7-614e-479e-8793-ba2d1d9b6427" containerName="payload-pre-processing"
Nov 03 19:20:50.097830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.097806    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cad21f-8b9e-4776-a66f-92ddc697acdc" containerName="revoke-keys"
Nov 03 19:20:50.102099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.102073    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.155192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.155161    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"]
Nov 03 19:20:50.196012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.195885    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:20:50.196012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.195967    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:20:50.196012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.195985    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:20:50.196012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.196015    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:20:50.215030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.215000    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vws2d\" (UniqueName: \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.215241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.215152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.215241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.215219    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.317059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.317024    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vws2d\" (UniqueName: \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.317539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.317107    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.317539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.317268    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.317804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.317752    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.319711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.319686    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.374722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.374652    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vws2d\" (UniqueName: \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.421994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.421965    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:50.807215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:50.807176    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"]
Nov 03 19:20:50.811856 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:20:50.811829    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db9a68d_a1e4_4ad8_aeec_54c287f75e40.slice/crio-15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f WatchSource:0}: Error finding container 15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f: Status 404 returned error can't find the container with id 15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f
Nov 03 19:20:51.025344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.025303    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:20:51.402756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.402675    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx" event={"ID":"9db9a68d-a1e4-4ad8-aeec-54c287f75e40","Type":"ContainerStarted","Data":"4bc87290ad000f5e9dd3b5650fbdd8fdc7cb56895f0e280ea9c3d398763d8396"}
Nov 03 19:20:51.402756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.402731    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx" event={"ID":"9db9a68d-a1e4-4ad8-aeec-54c287f75e40","Type":"ContainerStarted","Data":"15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f"}
Nov 03 19:20:51.512318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.512259    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx" podStartSLOduration=1.512242262 podStartE2EDuration="1.512242262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:20:51.510129852 +0000 UTC m=+2149.787483434" watchObservedRunningTime="2026-11-03 19:20:51.512242262 +0000 UTC m=+2149.789595877"
Nov 03 19:20:51.607288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.607257    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:20:51.737167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737083    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737168    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737201    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737235    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737296    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737363    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:51.737561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737398    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737460    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737562    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.737717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.737607    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\") pod \"b87517ec-5657-4fa7-85d6-d587ef05ba72\" (UID: \"b87517ec-5657-4fa7-85d6-d587ef05ba72\") "
Nov 03 19:20:51.738323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.738295    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.738955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.738927    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:51.739322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.739090    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:51.739322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.739158    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:51.739322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.739172    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:51.740107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.740080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:51.741548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.741520    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "kube-api-access-mprwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:51.741948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.741921    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:20:51.742508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.742487    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy" pod "b87517ec-5657-4fa7-85d6-d587ef05ba72" (UID: "b87517ec-5657-4fa7-85d6-d587ef05ba72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:51.840258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840223    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b87517ec-5657-4fa7-85d6-d587ef05ba72-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840259    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840271    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840280    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840288    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840297    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840307    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b87517ec-5657-4fa7-85d6-d587ef05ba72-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:51.840852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:51.840321    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mprwc\" (UniqueName: \"kubernetes.io/projected/b87517ec-5657-4fa7-85d6-d587ef05ba72-kube-api-access-mprwc\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:52.411659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.411622    4944 generic.go:356] "Generic (PLEG): container finished" podID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerID="89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab" exitCode=0
Nov 03 19:20:52.412095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.411679    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"
Nov 03 19:20:52.412095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.411710    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" event={"ID":"b87517ec-5657-4fa7-85d6-d587ef05ba72","Type":"ContainerDied","Data":"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"}
Nov 03 19:20:52.412095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.411772    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2" event={"ID":"b87517ec-5657-4fa7-85d6-d587ef05ba72","Type":"ContainerDied","Data":"8dd875f7c76f6e0ee63ad57dd17de7c0d65e1963296bb60065bc0bb76653f9e2"}
Nov 03 19:20:52.412095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.411813    4944 scope.go:122] "RemoveContainer" containerID="89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"
Nov 03 19:20:52.413568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.413540    4944 generic.go:356] "Generic (PLEG): container finished" podID="9db9a68d-a1e4-4ad8-aeec-54c287f75e40" containerID="4bc87290ad000f5e9dd3b5650fbdd8fdc7cb56895f0e280ea9c3d398763d8396" exitCode=0
Nov 03 19:20:52.413684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.413607    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx" event={"ID":"9db9a68d-a1e4-4ad8-aeec-54c287f75e40","Type":"ContainerDied","Data":"4bc87290ad000f5e9dd3b5650fbdd8fdc7cb56895f0e280ea9c3d398763d8396"}
Nov 03 19:20:52.422380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.422362    4944 scope.go:122] "RemoveContainer" containerID="89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"
Nov 03 19:20:52.422683 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:20:52.422662    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab\": container with ID starting with 89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab not found: ID does not exist" containerID="89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"
Nov 03 19:20:52.422753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.422689    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab"} err="failed to get container status \"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab\": rpc error: code = NotFound desc = could not find container \"89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab\": container with ID starting with 89aa9c596144c78434a1295232fe4a4c32b7f17ad45c22700b6e40588e913aab not found: ID does not exist"
Nov 03 19:20:52.675437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.675353    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:20:52.720979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:52.720948    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-hgkl2"]
Nov 03 19:20:53.571840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.571809    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:53.767361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.767330    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\" (UniqueName: \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\") pod \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") "
Nov 03 19:20:53.767559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.767394    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\") pod \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") "
Nov 03 19:20:53.767559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.767412    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\") pod \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\" (UID: \"9db9a68d-a1e4-4ad8-aeec-54c287f75e40\") "
Nov 03 19:20:53.767753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.767730    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca" pod "9db9a68d-a1e4-4ad8-aeec-54c287f75e40" (UID: "9db9a68d-a1e4-4ad8-aeec-54c287f75e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:53.767892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.767875    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:53.769338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.769314    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca" pod "9db9a68d-a1e4-4ad8-aeec-54c287f75e40" (UID: "9db9a68d-a1e4-4ad8-aeec-54c287f75e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:20:53.769438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.769335    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d" pod "9db9a68d-a1e4-4ad8-aeec-54c287f75e40" (UID: "9db9a68d-a1e4-4ad8-aeec-54c287f75e40"). InnerVolumeSpecName "kube-api-access-vws2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:53.868676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.868644    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vws2d\" (UniqueName: \"kubernetes.io/projected/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-kube-api-access-vws2d\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:53.868676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.868673    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9db9a68d-a1e4-4ad8-aeec-54c287f75e40-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:53.935619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:53.935590    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" path="/var/lib/kubelet/pods/b87517ec-5657-4fa7-85d6-d587ef05ba72/volumes"
Nov 03 19:20:54.429555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.429522    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"
Nov 03 19:20:54.429555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.429546    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx" event={"ID":"9db9a68d-a1e4-4ad8-aeec-54c287f75e40","Type":"ContainerDied","Data":"15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f"}
Nov 03 19:20:54.429812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.429628    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15f0279a029e56381b0819bfd03c8e25df47f4a5453cefa9ec169572371a360f"
Nov 03 19:20:54.431470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.431441    4944 generic.go:356] "Generic (PLEG): container finished" podID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerID="7b0b0f2697c4b2b453d66e23968da2f7a14120eaa9063d40dce5a0dc0b861589" exitCode=0
Nov 03 19:20:54.431602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.431499    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" event={"ID":"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e","Type":"ContainerDied","Data":"7b0b0f2697c4b2b453d66e23968da2f7a14120eaa9063d40dce5a0dc0b861589"}
Nov 03 19:20:54.647111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.647075    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:20:54.678173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678146    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678204    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678245    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678317    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678373    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678406    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.678632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678552    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.678696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678601    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.678756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678702    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678745    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:54.678888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678749    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678812    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.678888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.678849    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\") pod \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\" (UID: \"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e\") "
Nov 03 19:20:54.679098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679074    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.679635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679611    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.679719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679641    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.679719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679658    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.679719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679672    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.679719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.679685    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.680938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.680914    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:54.681218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.681129    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:20:54.681218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.681131    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.681218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.681196    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89" pod "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" (UID: "cc76a066-aca4-4ce2-8bfe-4f9d06cae22e"). InnerVolumeSpecName "kube-api-access-ztr89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:54.703016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.702990    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:20:54.781107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781075    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781196    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781247    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781272    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781299    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781355    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781402    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.781526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781424    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781505    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781620    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\") pod \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\" (UID: \"5fbab8f6-be89-4a17-b9e7-be736c42fb0e\") "
Nov 03 19:20:54.781857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.781831    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.782081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782032    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.782349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782323    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:20:54.782516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782495    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782524    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782540    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782556    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782569    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782583    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782597    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztr89\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-kube-api-access-ztr89\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782613    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.782973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.782674    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.784248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.784220    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:54.784537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.784510    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:20:54.784537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.784522    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "kube-api-access-lsbrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:20:54.784968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.784946    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy" pod "5fbab8f6-be89-4a17-b9e7-be736c42fb0e" (UID: "5fbab8f6-be89-4a17-b9e7-be736c42fb0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:20:54.884508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.884427    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.884508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.884470    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.884508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.884487    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.884508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.884501    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsbrg\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-kube-api-access-lsbrg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:54.884855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:54.884515    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbab8f6-be89-4a17-b9e7-be736c42fb0e-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:20:55.439965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.439932    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"
Nov 03 19:20:55.439965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.439950    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m" event={"ID":"cc76a066-aca4-4ce2-8bfe-4f9d06cae22e","Type":"ContainerDied","Data":"c81054893b4398be0ddff38867635b62f737fb9ff72df224a93c5947c2b0d361"}
Nov 03 19:20:55.440221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.440032    4944 scope.go:122] "RemoveContainer" containerID="7b0b0f2697c4b2b453d66e23968da2f7a14120eaa9063d40dce5a0dc0b861589"
Nov 03 19:20:55.441357 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.441333    4944 generic.go:356] "Generic (PLEG): container finished" podID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerID="1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2" exitCode=0
Nov 03 19:20:55.441490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.441386    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"
Nov 03 19:20:55.441490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.441427    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" event={"ID":"5fbab8f6-be89-4a17-b9e7-be736c42fb0e","Type":"ContainerDied","Data":"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"}
Nov 03 19:20:55.441490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.441467    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn" event={"ID":"5fbab8f6-be89-4a17-b9e7-be736c42fb0e","Type":"ContainerDied","Data":"8696b71315df84a28c91636c61c13a4fe8d911694905c7d0aa6f714079061af0"}
Nov 03 19:20:55.450216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.450195    4944 scope.go:122] "RemoveContainer" containerID="1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"
Nov 03 19:20:55.462212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.462190    4944 scope.go:122] "RemoveContainer" containerID="1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"
Nov 03 19:20:55.462682 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:20:55.462657    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2\": container with ID starting with 1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2 not found: ID does not exist" containerID="1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"
Nov 03 19:20:55.462771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.462691    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2"} err="failed to get container status \"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2\": rpc error: code = NotFound desc = could not find container \"1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2\": container with ID starting with 1baa27e71a8db072117a6e12cb3ed6919994ccdb3586d53c0e06a2e2efbba5a2 not found: ID does not exist"
Nov 03 19:20:55.595039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.595003    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:20:55.641224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.641191    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-lpw4m"]
Nov 03 19:20:55.700626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.700487    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:20:55.705597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.705568    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-smrsn"]
Nov 03 19:20:55.936948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.936915    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" path="/var/lib/kubelet/pods/5fbab8f6-be89-4a17-b9e7-be736c42fb0e/volumes"
Nov 03 19:20:55.937531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:55.937505    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" path="/var/lib/kubelet/pods/cc76a066-aca4-4ce2-8bfe-4f9d06cae22e/volumes"
Nov 03 19:20:56.253996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:20:56.253960    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:00.911038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:00.911004    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:02.831517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:02.831478    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:21:02.930303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:02.930267    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:21:03.281108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:03.281074    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:03.532387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:03.532303    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:03.808285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:03.808199    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:04.346429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:04.346392    4944 scope.go:122] "RemoveContainer" containerID="d195594067ab1ca8f06d140a2fb1ea0235c2fafe31fcc0b721b9d60889803f68"
Nov 03 19:21:04.356296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:04.356269    4944 scope.go:122] "RemoveContainer" containerID="047398e3964510ed24437763825c5b7713b35b55e3db2ba7f7d30111be59dfd9"
Nov 03 19:21:05.000814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:05.000479    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:05.225625 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:05.225590    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:21:05.704933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:05.704897    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:05.955908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:05.955810    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:06.204267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:06.204207    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:07.083031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:07.082998    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:07.334137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:07.334058    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:07.832836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:07.832775    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" podUID="fce11898-cc94-4875-9d78-4877672a3b72" containerName="payload-processing" containerID="cri-o://562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995" gracePeriod=25
Nov 03 19:21:07.931069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:07.931001    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" podUID="75e139db-8e50-4f7e-a08d-c998302b9f49" containerName="payload-pre-processing" containerID="cri-o://1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71" gracePeriod=25
Nov 03 19:21:08.118073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.118042    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:21:08.201984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.201956    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\" (UniqueName: \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\") pod \"fce11898-cc94-4875-9d78-4877672a3b72\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") "
Nov 03 19:21:08.202126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.202056    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\") pod \"fce11898-cc94-4875-9d78-4877672a3b72\" (UID: \"fce11898-cc94-4875-9d78-4877672a3b72\") "
Nov 03 19:21:08.202466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.202439    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume" pod "fce11898-cc94-4875-9d78-4877672a3b72" (UID: "fce11898-cc94-4875-9d78-4877672a3b72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:08.204263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.204240    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl" pod "fce11898-cc94-4875-9d78-4877672a3b72" (UID: "fce11898-cc94-4875-9d78-4877672a3b72"). InnerVolumeSpecName "kube-api-access-hn7gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:08.215186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.215166    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:21:08.303102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303072    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\") pod \"75e139db-8e50-4f7e-a08d-c998302b9f49\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") "
Nov 03 19:21:08.303261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303209    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\" (UniqueName: \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\") pod \"75e139db-8e50-4f7e-a08d-c998302b9f49\" (UID: \"75e139db-8e50-4f7e-a08d-c998302b9f49\") "
Nov 03 19:21:08.303467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303442    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume" pod "75e139db-8e50-4f7e-a08d-c998302b9f49" (UID: "75e139db-8e50-4f7e-a08d-c998302b9f49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:08.303742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303725    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce11898-cc94-4875-9d78-4877672a3b72-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:08.303841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303743    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e139db-8e50-4f7e-a08d-c998302b9f49-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:08.303841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.303753    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7gl\" (UniqueName: \"kubernetes.io/projected/fce11898-cc94-4875-9d78-4877672a3b72-kube-api-access-hn7gl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:08.305330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.305307    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs" pod "75e139db-8e50-4f7e-a08d-c998302b9f49" (UID: "75e139db-8e50-4f7e-a08d-c998302b9f49"). InnerVolumeSpecName "kube-api-access-cdkxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:08.404962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.404882    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdkxs\" (UniqueName: \"kubernetes.io/projected/75e139db-8e50-4f7e-a08d-c998302b9f49-kube-api-access-cdkxs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:08.567820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.567758    4944 generic.go:356] "Generic (PLEG): container finished" podID="fce11898-cc94-4875-9d78-4877672a3b72" containerID="562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995" exitCode=0
Nov 03 19:21:08.568018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.567833    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"
Nov 03 19:21:08.568018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.567839    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" event={"ID":"fce11898-cc94-4875-9d78-4877672a3b72","Type":"ContainerDied","Data":"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"}
Nov 03 19:21:08.568018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.567889    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz" event={"ID":"fce11898-cc94-4875-9d78-4877672a3b72","Type":"ContainerDied","Data":"e366841ebb32e9d6858cb70a1ceb57ce160a2eda679bc750d12755278d4ebc27"}
Nov 03 19:21:08.568018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.567910    4944 scope.go:122] "RemoveContainer" containerID="562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"
Nov 03 19:21:08.569296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.569276    4944 generic.go:356] "Generic (PLEG): container finished" podID="75e139db-8e50-4f7e-a08d-c998302b9f49" containerID="1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71" exitCode=0
Nov 03 19:21:08.569400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.569306    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"
Nov 03 19:21:08.569400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.569355    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" event={"ID":"75e139db-8e50-4f7e-a08d-c998302b9f49","Type":"ContainerDied","Data":"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"}
Nov 03 19:21:08.569483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.569406    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k" event={"ID":"75e139db-8e50-4f7e-a08d-c998302b9f49","Type":"ContainerDied","Data":"be3f2f19c6aa74c12e43cb4d12a59757aa519f25e5092bbd2a24743bb6eb0181"}
Nov 03 19:21:08.578771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.578754    4944 scope.go:122] "RemoveContainer" containerID="562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"
Nov 03 19:21:08.579104 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:08.579079    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995\": container with ID starting with 562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995 not found: ID does not exist" containerID="562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"
Nov 03 19:21:08.579212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.579113    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995"} err="failed to get container status \"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995\": rpc error: code = NotFound desc = could not find container \"562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995\": container with ID starting with 562d975f43897f6cf59bd4ed42d81d388d0476acf6b604a6e89e1ce00c3e7995 not found: ID does not exist"
Nov 03 19:21:08.579212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.579134    4944 scope.go:122] "RemoveContainer" containerID="1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"
Nov 03 19:21:08.587961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.587945    4944 scope.go:122] "RemoveContainer" containerID="1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"
Nov 03 19:21:08.588227 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:08.588209    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71\": container with ID starting with 1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71 not found: ID does not exist" containerID="1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"
Nov 03 19:21:08.588291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.588237    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71"} err="failed to get container status \"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71\": rpc error: code = NotFound desc = could not find container \"1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71\": container with ID starting with 1f81c5d7dae03e7d5cfe48ca28a854131401a4921c8cbc1b1a1a1c5398878b71 not found: ID does not exist"
Nov 03 19:21:08.721305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.721219    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:21:08.726133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.726110    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-144890-6d59d5996d-k2xxz"]
Nov 03 19:21:08.828476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.828444    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:21:08.833312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:08.833284    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-144890-7f4fc7586c-8rl9k"]
Nov 03 19:21:09.936409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:09.936370    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e139db-8e50-4f7e-a08d-c998302b9f49" path="/var/lib/kubelet/pods/75e139db-8e50-4f7e-a08d-c998302b9f49/volumes"
Nov 03 19:21:09.937147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:09.937125    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce11898-cc94-4875-9d78-4877672a3b72" path="/var/lib/kubelet/pods/fce11898-cc94-4875-9d78-4877672a3b72/volumes"
Nov 03 19:21:10.973577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:10.973542    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:11.610595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:11.610555    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:11.864973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:11.864823    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:14.456657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:14.456591    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:14.892361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:14.892324    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:15.492272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:15.492233    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:15.749114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:15.749041    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:15.996318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:15.996281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:16.721863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:16.721823    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:16.974774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:16.974690    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:17.307263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:17.307226    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:17.592868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:17.592713    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:17.716528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:17.716488    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"]
Nov 03 19:21:17.763002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:17.762967    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-08baeb-54vjm"]
Nov 03 19:21:17.935757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:17.935659    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022200f3-69d6-4084-96ca-31f3cdd59294" path="/var/lib/kubelet/pods/022200f3-69d6-4084-96ca-31f3cdd59294/volumes"
Nov 03 19:21:18.237266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:18.237134    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:18.550755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:18.550720    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:18.810237 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:18.810151    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:19.060487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:19.060357    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:19.072343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:19.072308    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"]
Nov 03 19:21:19.117516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:19.117485    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-144890-jlvqx"]
Nov 03 19:21:19.521236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:19.521203    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:19.934894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:19.934784    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db9a68d-a1e4-4ad8-aeec-54c287f75e40" path="/var/lib/kubelet/pods/9db9a68d-a1e4-4ad8-aeec-54c287f75e40/volumes"
Nov 03 19:21:21.411817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:21.411525    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:21.766759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:21.766710    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:24.019731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:24.019669    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:24.862913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:24.862818    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:25.120547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:25.120466    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:25.383592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:25.383457    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:25.660724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:25.660621    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:26.045195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045155    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"]
Nov 03 19:21:26.045941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045917    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy"
Nov 03 19:21:26.045941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045936    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045975    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9db9a68d-a1e4-4ad8-aeec-54c287f75e40" containerName="revoke-keys"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045980    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db9a68d-a1e4-4ad8-aeec-54c287f75e40" containerName="revoke-keys"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.045999    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046005    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046014    4944 cpu_manager.go:414] "containerMap: removing container" podUID="75e139db-8e50-4f7e-a08d-c998302b9f49" containerName="payload-pre-processing"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046019    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e139db-8e50-4f7e-a08d-c998302b9f49" containerName="payload-pre-processing"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046029    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fce11898-cc94-4875-9d78-4877672a3b72" containerName="payload-processing"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046033    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce11898-cc94-4875-9d78-4877672a3b72" containerName="payload-processing"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046042    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy"
Nov 03 19:21:26.046122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046047    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046141    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce11898-cc94-4875-9d78-4877672a3b72" containerName="payload-processing"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046149    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbab8f6-be89-4a17-b9e7-be736c42fb0e" containerName="istio-proxy"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046156    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b87517ec-5657-4fa7-85d6-d587ef05ba72" containerName="istio-proxy"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046169    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db9a68d-a1e4-4ad8-aeec-54c287f75e40" containerName="revoke-keys"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046178    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc76a066-aca4-4ce2-8bfe-4f9d06cae22e" containerName="istio-proxy"
Nov 03 19:21:26.046438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.046185    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e139db-8e50-4f7e-a08d-c998302b9f49" containerName="payload-pre-processing"
Nov 03 19:21:26.049495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.049478    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.074785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.074745    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.074994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.074922    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdgt\" (UniqueName: \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.074994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.074971    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.103599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.103562    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"]
Nov 03 19:21:26.145384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.145350    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:21:26.145923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.145439    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:21:26.145923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.145453    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:21:26.146096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.145459    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:21:26.176530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.176499    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.176798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.176769    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.176880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.176866    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdgt\" (UniqueName: \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.177318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.177285    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.179443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.179426    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.234430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.234406    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdgt\" (UniqueName: \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\") pod \"maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.363614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.363527    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:26.748844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:26.748783    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"]
Nov 03 19:21:26.754303 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:21:26.754269    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode324dc9c_05f9_4ba4_9738_bdea05154325.slice/crio-68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d WatchSource:0}: Error finding container 68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d: Status 404 returned error can't find the container with id 68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d
Nov 03 19:21:27.092365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.092285    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:27.261407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.261369    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:21:27.261819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.261619    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy" containerID="cri-o://150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456" gracePeriod=30
Nov 03 19:21:27.421469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.421373    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:27.688671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.688578    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:27.730302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.730276    4944 generic.go:356] "Generic (PLEG): container finished" podID="e324dc9c-05f9-4ba4-9738-bdea05154325" containerID="bfe760fa8feced3aa40326748ac90bca4b28d7a3f94fc5e34ad95f75b3763edd" exitCode=0
Nov 03 19:21:27.730436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.730330    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh" event={"ID":"e324dc9c-05f9-4ba4-9738-bdea05154325","Type":"ContainerDied","Data":"bfe760fa8feced3aa40326748ac90bca4b28d7a3f94fc5e34ad95f75b3763edd"}
Nov 03 19:21:27.730499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.730431    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh" event={"ID":"e324dc9c-05f9-4ba4-9738-bdea05154325","Type":"ContainerStarted","Data":"68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d"}
Nov 03 19:21:27.901248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.901211    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"]
Nov 03 19:21:27.905562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.905541    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:27.928096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.928069    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkwx\" (UniqueName: \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:27.928281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.928261    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:27.928415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.928397    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:27.964555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:27.964475    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"]
Nov 03 19:21:28.029677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.029641    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkwx\" (UniqueName: \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.029885 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.029779    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.029981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.029960    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.030578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.030553    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.032302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.032278    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.098922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.098885    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkwx\" (UniqueName: \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.219169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.219083    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:28.598242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.598210    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"]
Nov 03 19:21:28.603239 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:21:28.603214    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b61f632_c45e_4ad8_89bb_7d8d1b9caa7b.slice/crio-cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6 WatchSource:0}: Error finding container cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6: Status 404 returned error can't find the container with id cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6
Nov 03 19:21:28.738866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.738831    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk" event={"ID":"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b","Type":"ContainerStarted","Data":"cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6"}
Nov 03 19:21:28.887422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.887399    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:28.941121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.941095    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\") pod \"e324dc9c-05f9-4ba4-9738-bdea05154325\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") "
Nov 03 19:21:28.941272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.941189    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\") pod \"e324dc9c-05f9-4ba4-9738-bdea05154325\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") "
Nov 03 19:21:28.941272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.941257    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\" (UniqueName: \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\") pod \"e324dc9c-05f9-4ba4-9738-bdea05154325\" (UID: \"e324dc9c-05f9-4ba4-9738-bdea05154325\") "
Nov 03 19:21:28.941537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.941513    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca" pod "e324dc9c-05f9-4ba4-9738-bdea05154325" (UID: "e324dc9c-05f9-4ba4-9738-bdea05154325"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:28.941822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.941805    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:28.943099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.943080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca" pod "e324dc9c-05f9-4ba4-9738-bdea05154325" (UID: "e324dc9c-05f9-4ba4-9738-bdea05154325"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:21:28.943205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:28.943184    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt" pod "e324dc9c-05f9-4ba4-9738-bdea05154325" (UID: "e324dc9c-05f9-4ba4-9738-bdea05154325"). InnerVolumeSpecName "kube-api-access-xtdgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:29.043062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.043024    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdgt\" (UniqueName: \"kubernetes.io/projected/e324dc9c-05f9-4ba4-9738-bdea05154325-kube-api-access-xtdgt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:29.043062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.043058    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e324dc9c-05f9-4ba4-9738-bdea05154325-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:29.312428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.312398    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:29.585529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.585494    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:29.747315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.747282    4944 generic.go:356] "Generic (PLEG): container finished" podID="6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" containerID="a373676049d061222268fc937c23447f02bc7ba90630acf07b8d6c4f9da234cf" exitCode=0
Nov 03 19:21:29.747712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.747321    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk" event={"ID":"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b","Type":"ContainerDied","Data":"a373676049d061222268fc937c23447f02bc7ba90630acf07b8d6c4f9da234cf"}
Nov 03 19:21:29.748876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.748854    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh" event={"ID":"e324dc9c-05f9-4ba4-9738-bdea05154325","Type":"ContainerDied","Data":"68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d"}
Nov 03 19:21:29.748876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.748874    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"
Nov 03 19:21:29.749059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:29.748888    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68efba9f6c52f4625948ad82dfa6d1e1e3588762fc7f393786fbb5cff502280d"
Nov 03 19:21:30.916652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.916628    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:30.968379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.968350    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\" (UniqueName: \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\") pod \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") "
Nov 03 19:21:30.968559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.968444    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\") pod \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") "
Nov 03 19:21:30.968559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.968468    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\") pod \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\" (UID: \"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b\") "
Nov 03 19:21:30.969035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.968990    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca" pod "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" (UID: "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:30.969418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.969392    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:30.970833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.970779    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca" pod "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" (UID: "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:21:30.971023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:30.971002    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx" pod "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" (UID: "6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b"). InnerVolumeSpecName "kube-api-access-2mkwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:31.070806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:31.070705    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkwx\" (UniqueName: \"kubernetes.io/projected/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-kube-api-access-2mkwx\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:31.070806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:31.070740    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:31.764650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:31.764619    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"
Nov 03 19:21:31.764844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:31.764654    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk" event={"ID":"6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b","Type":"ContainerDied","Data":"cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6"}
Nov 03 19:21:31.764844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:31.764709    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf1a120bb6db5dd297d3446bfe548beb39aad2b13bad2c13f22f1e34a61f02c6"
Nov 03 19:21:32.584929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.584903    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:21:32.690944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.690913    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.690983    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691016    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691042    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691176    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691207    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691227    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:32.691325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691263    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691330    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691350    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:32.691423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691395    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\") pod \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\" (UID: \"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1\") "
Nov 03 19:21:32.691596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691573    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:32.691658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691594    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:32.691711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.691682    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:32.692115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.692097    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.692191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.692119    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.692191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.692132    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.692191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.692140    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.692191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.692149    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.693679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.693653    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:21:32.693679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.693664    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:21:32.693896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.693825    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:32.693896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.693849    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54" pod "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" (UID: "cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1"). InnerVolumeSpecName "kube-api-access-sxv54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:32.773874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.773836    4944 generic.go:356] "Generic (PLEG): container finished" podID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerID="150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456" exitCode=0
Nov 03 19:21:32.774047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.773886    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"
Nov 03 19:21:32.774047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.773912    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerDied","Data":"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"}
Nov 03 19:21:32.774047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.773984    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6" event={"ID":"cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1","Type":"ContainerDied","Data":"95c54bec17052301bfc10fa18a048df2a05b251cb207467c6926bb5f70809f4f"}
Nov 03 19:21:32.774047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.774014    4944 scope.go:122] "RemoveContainer" containerID="150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"
Nov 03 19:21:32.788157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.788136    4944 scope.go:122] "RemoveContainer" containerID="cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"
Nov 03 19:21:32.793831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.793803    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxv54\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-kube-api-access-sxv54\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.793831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.793832    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.794008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.793842    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.794008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.793853    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:32.798152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.798138    4944 scope.go:122] "RemoveContainer" containerID="150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"
Nov 03 19:21:32.798420 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:32.798400    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456\": container with ID starting with 150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456 not found: ID does not exist" containerID="150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"
Nov 03 19:21:32.798494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.798428    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456"} err="failed to get container status \"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456\": rpc error: code = NotFound desc = could not find container \"150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456\": container with ID starting with 150789d8dcb2b4308c3b4933d809107c41404ad56a23b865569357ed03c7f456 not found: ID does not exist"
Nov 03 19:21:32.798494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.798446    4944 scope.go:122] "RemoveContainer" containerID="cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"
Nov 03 19:21:32.798730 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:32.798709    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d\": container with ID starting with cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d not found: ID does not exist" containerID="cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"
Nov 03 19:21:32.798775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.798740    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d"} err="failed to get container status \"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d\": rpc error: code = NotFound desc = could not find container \"cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d\": container with ID starting with cb2c7170d1f79756cde999f0554be7dee1cd9cae7c77bb3e7260b30bfdf6fd6d not found: ID does not exist"
Nov 03 19:21:32.930559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.930471    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:21:32.934507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:32.934460    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-144890-openshift-default-86b88f58cf-5b6h6"]
Nov 03 19:21:33.936706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:33.936668    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" path="/var/lib/kubelet/pods/cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1/volumes"
Nov 03 19:21:34.423998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:34.423959    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:34.673264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:34.673222    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:35.224925 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:35.224889    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:21:38.863901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:38.863845    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:39.112766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:39.112732    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:39.495862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:39.495759    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:41.274770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:41.274692    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:21:41.371759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:41.371725    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:21:41.376766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:41.376736    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:41.631870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:41.631766    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:41.874969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:41.874940    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:44.592367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:44.592307    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:46.275492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.275431    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" podUID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" containerName="payload-processing" containerID="cri-o://9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48" gracePeriod=25
Nov 03 19:21:46.372672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.372634    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" podUID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" containerName="payload-pre-processing" containerID="cri-o://158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359" gracePeriod=25
Nov 03 19:21:46.549003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.548962    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:21:46.594226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.594192    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\") pod \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") "
Nov 03 19:21:46.594502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.594481    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\" (UniqueName: \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\") pod \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\" (UID: \"73c927f8-0b3d-4d64-9a7b-5b4172be113d\") "
Nov 03 19:21:46.594638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.594617    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume" pod "73c927f8-0b3d-4d64-9a7b-5b4172be113d" (UID: "73c927f8-0b3d-4d64-9a7b-5b4172be113d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:46.594986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.594963    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c927f8-0b3d-4d64-9a7b-5b4172be113d-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:46.598029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.598005    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv" pod "73c927f8-0b3d-4d64-9a7b-5b4172be113d" (UID: "73c927f8-0b3d-4d64-9a7b-5b4172be113d"). InnerVolumeSpecName "kube-api-access-mrqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:46.639171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.639148    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:21:46.695882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.695851    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\") pod \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") "
Nov 03 19:21:46.696064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.695943    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\" (UniqueName: \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\") pod \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\" (UID: \"71ac16fa-1889-4ee2-954f-3a8a89c158a6\") "
Nov 03 19:21:46.696334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.696304    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume" pod "71ac16fa-1889-4ee2-954f-3a8a89c158a6" (UID: "71ac16fa-1889-4ee2-954f-3a8a89c158a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:21:46.696423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.696341    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrqsv\" (UniqueName: \"kubernetes.io/projected/73c927f8-0b3d-4d64-9a7b-5b4172be113d-kube-api-access-mrqsv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:46.698155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.698125    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m" pod "71ac16fa-1889-4ee2-954f-3a8a89c158a6" (UID: "71ac16fa-1889-4ee2-954f-3a8a89c158a6"). InnerVolumeSpecName "kube-api-access-gmb5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:21:46.797780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.797708    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmb5m\" (UniqueName: \"kubernetes.io/projected/71ac16fa-1889-4ee2-954f-3a8a89c158a6-kube-api-access-gmb5m\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:46.797780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.797737    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71ac16fa-1889-4ee2-954f-3a8a89c158a6-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:21:46.877973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.877938    4944 generic.go:356] "Generic (PLEG): container finished" podID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" containerID="158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359" exitCode=0
Nov 03 19:21:46.878168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.877993    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"
Nov 03 19:21:46.878168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.878029    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" event={"ID":"71ac16fa-1889-4ee2-954f-3a8a89c158a6","Type":"ContainerDied","Data":"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"}
Nov 03 19:21:46.878168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.878091    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z" event={"ID":"71ac16fa-1889-4ee2-954f-3a8a89c158a6","Type":"ContainerDied","Data":"54f89ed4b5248275c2e329c63ebb8b39da2e50d7b831ac525df6fa759ef964ca"}
Nov 03 19:21:46.878168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.878119    4944 scope.go:122] "RemoveContainer" containerID="158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"
Nov 03 19:21:46.879660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.879633    4944 generic.go:356] "Generic (PLEG): container finished" podID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" containerID="9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48" exitCode=0
Nov 03 19:21:46.879775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.879676    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"
Nov 03 19:21:46.879928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.879770    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" event={"ID":"73c927f8-0b3d-4d64-9a7b-5b4172be113d","Type":"ContainerDied","Data":"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"}
Nov 03 19:21:46.879928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.879862    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58" event={"ID":"73c927f8-0b3d-4d64-9a7b-5b4172be113d","Type":"ContainerDied","Data":"c5a93aadb390e0c72a8c295929f4613d6b0fb0bbdefb9024c0d25ae583afe96a"}
Nov 03 19:21:46.889365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.889335    4944 scope.go:122] "RemoveContainer" containerID="158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"
Nov 03 19:21:46.889716 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:46.889665    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359\": container with ID starting with 158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359 not found: ID does not exist" containerID="158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"
Nov 03 19:21:46.889716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.889696    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359"} err="failed to get container status \"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359\": rpc error: code = NotFound desc = could not find container \"158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359\": container with ID starting with 158b728509349a826a65a3e9e44d9f7343cdcc861d6f2917ad6c7490912cf359 not found: ID does not exist"
Nov 03 19:21:46.889716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.889714    4944 scope.go:122] "RemoveContainer" containerID="9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"
Nov 03 19:21:46.900018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.899995    4944 scope.go:122] "RemoveContainer" containerID="9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"
Nov 03 19:21:46.900296 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:21:46.900277    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48\": container with ID starting with 9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48 not found: ID does not exist" containerID="9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"
Nov 03 19:21:46.900361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:46.900304    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48"} err="failed to get container status \"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48\": rpc error: code = NotFound desc = could not find container \"9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48\": container with ID starting with 9ecedcd4bbb8799b5c71f072123ac3a09ce2c172f6afb33c16d7419231eb6c48 not found: ID does not exist"
Nov 03 19:21:47.038047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.038013    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:21:47.043059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.043029    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-81a044-778fcb5d6c-m457z"]
Nov 03 19:21:47.160437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.160334    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:21:47.175429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.175390    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-81a044-5f5f47fccd-ltg58"]
Nov 03 19:21:47.935672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.935634    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" path="/var/lib/kubelet/pods/71ac16fa-1889-4ee2-954f-3a8a89c158a6/volumes"
Nov 03 19:21:47.936300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:47.936278    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" path="/var/lib/kubelet/pods/73c927f8-0b3d-4d64-9a7b-5b4172be113d/volumes"
Nov 03 19:21:49.462170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:49.462121    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:54.002773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:54.002735    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:55.399653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:55.399619    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:55.649471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:55.649440    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:21:58.677950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:21:58.677893    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:02.618942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:02.618902    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:04.085266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.085227    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:04.334258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.334218    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:04.462302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.462259    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"]
Nov 03 19:22:04.507989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.507957    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bbccf3-j87st"]
Nov 03 19:22:04.588640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.588608    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:04.887042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.887006    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:22:04.887264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.887235    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" containerID="cri-o://6ab447841868f6d2ae8e831a8c834609182f790dd5b21186ded3e7bd93e052a7" gracePeriod=30
Nov 03 19:22:04.973919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:04.973884    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:05.225079 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:22:05.225004    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:22:05.934961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:05.934924    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cad21f-8b9e-4776-a66f-92ddc697acdc" path="/var/lib/kubelet/pods/a3cad21f-8b9e-4776-a66f-92ddc697acdc/volumes"
Nov 03 19:22:06.146273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:06.146232    4944 patch_prober.go:29] interesting pod/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:22:06.146449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:06.146301    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:22:09.195910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:09.195875    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:10.049190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.049155    4944 generic.go:356] "Generic (PLEG): container finished" podID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerID="6ab447841868f6d2ae8e831a8c834609182f790dd5b21186ded3e7bd93e052a7" exitCode=0
Nov 03 19:22:10.049190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.049192    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" event={"ID":"68eb852d-fb20-4b7a-a105-12c26a7bcb38","Type":"ContainerDied","Data":"6ab447841868f6d2ae8e831a8c834609182f790dd5b21186ded3e7bd93e052a7"}
Nov 03 19:22:10.160552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.160486    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:22:10.219376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219346    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219392    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219422    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219441    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219469    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219512    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219545    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219585    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219730    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219747    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:10.219851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219760    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\") pod \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\" (UID: \"68eb852d-fb20-4b7a-a105-12c26a7bcb38\") "
Nov 03 19:22:10.220435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.219977    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:10.220435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220007    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:22:10.220435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220264    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:10.220435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220427    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.220615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220450    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.220615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220494    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.220615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220507    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.220615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.220522    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.221948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.221924    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:10.222061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.221984    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:22:10.222241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.222213    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:22:10.222241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.222230    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm" pod "68eb852d-fb20-4b7a-a105-12c26a7bcb38" (UID: "68eb852d-fb20-4b7a-a105-12c26a7bcb38"). InnerVolumeSpecName "kube-api-access-vc6wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:10.322199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.322155    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6wm\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-kube-api-access-vc6wm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.322199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.322198    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.322401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.322214    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:10.322401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:10.322227    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68eb852d-fb20-4b7a-a105-12c26a7bcb38-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:11.056481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.056449    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"
Nov 03 19:22:11.056481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.056467    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858" event={"ID":"68eb852d-fb20-4b7a-a105-12c26a7bcb38","Type":"ContainerDied","Data":"379663e38639d656a3dbe937b5917c4b12a12af67bbfc6c8019c1b2aee5f490b"}
Nov 03 19:22:11.056703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.056529    4944 scope.go:122] "RemoveContainer" containerID="6ab447841868f6d2ae8e831a8c834609182f790dd5b21186ded3e7bd93e052a7"
Nov 03 19:22:11.145840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.145802    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:11.260623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.260588    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:22:11.309747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.309648    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-4v858"]
Nov 03 19:22:11.392648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.392611    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:11.638495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.638410    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:11.937501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.937425    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" path="/var/lib/kubelet/pods/68eb852d-fb20-4b7a-a105-12c26a7bcb38/volumes"
Nov 03 19:22:11.937740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.937725    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"]
Nov 03 19:22:11.938264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938250    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" containerName="revoke-keys"
Nov 03 19:22:11.938307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938265    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" containerName="revoke-keys"
Nov 03 19:22:11.938307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938281    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:22:11.938307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938286    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:22:11.938307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938294    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e324dc9c-05f9-4ba4-9738-bdea05154325" containerName="revoke-keys"
Nov 03 19:22:11.938307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938299    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e324dc9c-05f9-4ba4-9738-bdea05154325" containerName="revoke-keys"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938315    4944 cpu_manager.go:414] "containerMap: removing container" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938321    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938338    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938344    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938360    4944 cpu_manager.go:414] "containerMap: removing container" podUID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" containerName="payload-pre-processing"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938364    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" containerName="payload-pre-processing"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938382    4944 cpu_manager.go:414] "containerMap: removing container" podUID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" containerName="payload-processing"
Nov 03 19:22:11.938462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938386    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" containerName="payload-processing"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938465    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ac16fa-1889-4ee2-954f-3a8a89c158a6" containerName="payload-pre-processing"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938474    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c927f8-0b3d-4d64-9a7b-5b4172be113d" containerName="payload-processing"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938481    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" containerName="revoke-keys"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938488    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938493    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e324dc9c-05f9-4ba4-9738-bdea05154325" containerName="revoke-keys"
Nov 03 19:22:11.938717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.938500    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="68eb852d-fb20-4b7a-a105-12c26a7bcb38" containerName="istio-proxy"
Nov 03 19:22:11.943345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.943327    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:11.995932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.995896    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:22:11.995932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.995914    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:22:11.998072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.998023    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:22:11.998202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.998074    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"]
Nov 03 19:22:11.998278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:11.998210    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:22:12.057127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.056824    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.057127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.057003    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.057362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.057255    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9vg\" (UniqueName: \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.159158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.159127    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.159347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.159189    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.159347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.159215    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9vg\" (UniqueName: \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.159832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.159775    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.161481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.161465    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.216167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.216107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9vg\" (UniqueName: \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\") pod \"maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.256466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.256439    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:12.628509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:12.628472    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"]
Nov 03 19:22:12.633612 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:22:12.633582    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e91da8_ec07_48a9_89fb_9d2cbfdb260f.slice/crio-7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4 WatchSource:0}: Error finding container 7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4: Status 404 returned error can't find the container with id 7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4
Nov 03 19:22:13.072137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:13.072101    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj" event={"ID":"18e91da8-ec07-48a9-89fb-9d2cbfdb260f","Type":"ContainerStarted","Data":"3abf39b8c21b293c9d6115b85b61e01919c45cd15cf05a84ae09a8159091dd80"}
Nov 03 19:22:13.072347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:13.072168    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj" event={"ID":"18e91da8-ec07-48a9-89fb-9d2cbfdb260f","Type":"ContainerStarted","Data":"7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4"}
Nov 03 19:22:13.227195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:13.227089    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj" podStartSLOduration=2.227075778 podStartE2EDuration="2.227075778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:22:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:22:13.179936348 +0000 UTC m=+2231.457289929" watchObservedRunningTime="2026-11-03 19:22:13.227075778 +0000 UTC m=+2231.504429361"
Nov 03 19:22:13.992054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:13.992023    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:14.080359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:14.080322    4944 generic.go:356] "Generic (PLEG): container finished" podID="18e91da8-ec07-48a9-89fb-9d2cbfdb260f" containerID="3abf39b8c21b293c9d6115b85b61e01919c45cd15cf05a84ae09a8159091dd80" exitCode=0
Nov 03 19:22:14.080541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:14.080409    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj" event={"ID":"18e91da8-ec07-48a9-89fb-9d2cbfdb260f","Type":"ContainerDied","Data":"3abf39b8c21b293c9d6115b85b61e01919c45cd15cf05a84ae09a8159091dd80"}
Nov 03 19:22:15.258642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.258616    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:15.399842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.399732    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\" (UniqueName: \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\") pod \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") "
Nov 03 19:22:15.399842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.399817    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\") pod \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") "
Nov 03 19:22:15.400060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.400005    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\") pod \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\" (UID: \"18e91da8-ec07-48a9-89fb-9d2cbfdb260f\") "
Nov 03 19:22:15.400432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.400397    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca" pod "18e91da8-ec07-48a9-89fb-9d2cbfdb260f" (UID: "18e91da8-ec07-48a9-89fb-9d2cbfdb260f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:22:15.400626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.400599    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:15.402505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.402478    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca" pod "18e91da8-ec07-48a9-89fb-9d2cbfdb260f" (UID: "18e91da8-ec07-48a9-89fb-9d2cbfdb260f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:22:15.402618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.402489    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg" pod "18e91da8-ec07-48a9-89fb-9d2cbfdb260f" (UID: "18e91da8-ec07-48a9-89fb-9d2cbfdb260f"). InnerVolumeSpecName "kube-api-access-7n9vg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:15.501647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.501614    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n9vg\" (UniqueName: \"kubernetes.io/projected/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-kube-api-access-7n9vg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:15.501647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:15.501646    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e91da8-ec07-48a9-89fb-9d2cbfdb260f-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:16.096983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:16.096954    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"
Nov 03 19:22:16.096983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:16.096985    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj" event={"ID":"18e91da8-ec07-48a9-89fb-9d2cbfdb260f","Type":"ContainerDied","Data":"7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4"}
Nov 03 19:22:16.097302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:16.097029    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7430afb43cc3bcaa561912e8ae207a61d6591218c8bd00c3a2b9acd5ae0785f4"
Nov 03 19:22:18.800570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:18.800529    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:23.474523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:23.474489    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:23.544658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:23.544626    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:22:23.997623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:23.997591    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:24.243630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:24.243597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:28.008069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.008033    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:28.545776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.545733    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" podUID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" containerName="payload-processing" containerID="cri-o://25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28" gracePeriod=25
Nov 03 19:22:28.810965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.810906    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:22:28.971109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.971067    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\") pod \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") "
Nov 03 19:22:28.971284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.971206    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\" (UniqueName: \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\") pod \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\" (UID: \"b9ca584f-24ea-49b6-9c0d-04d9d2150edd\") "
Nov 03 19:22:28.971500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.971470    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume" pod "b9ca584f-24ea-49b6-9c0d-04d9d2150edd" (UID: "b9ca584f-24ea-49b6-9c0d-04d9d2150edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:22:28.971780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.971760    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:28.973301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:28.973277    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc" pod "b9ca584f-24ea-49b6-9c0d-04d9d2150edd" (UID: "b9ca584f-24ea-49b6-9c0d-04d9d2150edd"). InnerVolumeSpecName "kube-api-access-6ctmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:22:29.073752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.073666    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctmc\" (UniqueName: \"kubernetes.io/projected/b9ca584f-24ea-49b6-9c0d-04d9d2150edd-kube-api-access-6ctmc\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:22:29.194604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.194574    4944 generic.go:356] "Generic (PLEG): container finished" podID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" containerID="25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28" exitCode=0
Nov 03 19:22:29.194818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.194617    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"
Nov 03 19:22:29.194818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.194655    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" event={"ID":"b9ca584f-24ea-49b6-9c0d-04d9d2150edd","Type":"ContainerDied","Data":"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"}
Nov 03 19:22:29.194818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.194713    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7" event={"ID":"b9ca584f-24ea-49b6-9c0d-04d9d2150edd","Type":"ContainerDied","Data":"38788e848e967cf52352dda5bd1e557084d36556d960bf5404cc22256939982b"}
Nov 03 19:22:29.194818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.194742    4944 scope.go:122] "RemoveContainer" containerID="25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"
Nov 03 19:22:29.205538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.205520    4944 scope.go:122] "RemoveContainer" containerID="25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"
Nov 03 19:22:29.205805 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:22:29.205774    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28\": container with ID starting with 25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28 not found: ID does not exist" containerID="25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"
Nov 03 19:22:29.205852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.205817    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28"} err="failed to get container status \"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28\": rpc error: code = NotFound desc = could not find container \"25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28\": container with ID starting with 25ff09985b93e500d1ac3f49807ba8f1df06ca0c8bdd38b543f5780e6dcb7e28 not found: ID does not exist"
Nov 03 19:22:29.351422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.351336    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:22:29.396931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.396899    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-1cc3f2-84559ff8cb-zvgm7"]
Nov 03 19:22:29.759313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.759281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:29.935406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:29.935370    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" path="/var/lib/kubelet/pods/b9ca584f-24ea-49b6-9c0d-04d9d2150edd/volumes"
Nov 03 19:22:32.420803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:32.420762    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:32.665981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:32.665945    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:35.225284 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:22:35.225252    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:22:37.359007 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:37.358960    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:41.801445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:41.801399    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:42.046963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:42.046929    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:42.287894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:42.287863    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:44.446076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:44.446036    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:44.693881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:44.693846    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:46.558954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:46.558923    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:51.363248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:51.363206    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:55.552486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:55.552449    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:56.055260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:56.055216    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:56.901709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:56.901674    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:57.193420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.193296    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:57.462950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.462864    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:57.655366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.655337    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"]
Nov 03 19:22:57.702046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.702010    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-81a044-8jtmk"]
Nov 03 19:22:57.711549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.711522    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:57.940969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:57.940926    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b" path="/var/lib/kubelet/pods/6b61f632-c45e-4ad8-89bb-7d8d1b9caa7b/volumes"
Nov 03 19:22:58.031177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:58.031145    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:22:58.282373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:22:58.282340    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:02.548997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.548966    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:02.789258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.789213    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"]
Nov 03 19:23:02.792250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792205    4944 cpu_manager.go:414] "containerMap: removing container" podUID="18e91da8-ec07-48a9-89fb-9d2cbfdb260f" containerName="revoke-keys"
Nov 03 19:23:02.792250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792235    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e91da8-ec07-48a9-89fb-9d2cbfdb260f" containerName="revoke-keys"
Nov 03 19:23:02.792504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792340    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" containerName="payload-processing"
Nov 03 19:23:02.792504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792355    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" containerName="payload-processing"
Nov 03 19:23:02.792865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792847    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ca584f-24ea-49b6-9c0d-04d9d2150edd" containerName="payload-processing"
Nov 03 19:23:02.792994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792879    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5bd611-ebd4-41fa-8ddc-cc519a63b0b1" containerName="istio-proxy"
Nov 03 19:23:02.792994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.792901    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e91da8-ec07-48a9-89fb-9d2cbfdb260f" containerName="revoke-keys"
Nov 03 19:23:02.797332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.797302    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.812890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.812768    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:02.836884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.836848    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"]
Nov 03 19:23:02.878648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.878610    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:23:02.878648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.878638    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:23:02.878648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.878650    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:23:02.892389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.892361    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.892509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.892426    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.892509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.892473    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qbdd\" (UniqueName: \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.895315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.895299    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:23:02.993878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.993845    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.994074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.993904    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.994074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.993927    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qbdd\" (UniqueName: \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.994563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.994542    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:02.996312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:02.996289    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:03.050831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:03.050800    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qbdd\" (UniqueName: \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\") pod \"maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:03.116201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:03.116121    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:03.489515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:03.489489    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"]
Nov 03 19:23:03.494917 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:23:03.494887    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636a0f54_e9de_4d2d_b2c5_ab350df01ae0.slice/crio-bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1 WatchSource:0}: Error finding container bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1: Status 404 returned error can't find the container with id bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1
Nov 03 19:23:03.512855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:03.512830    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44" event={"ID":"636a0f54-e9de-4d2d-b2c5-ab350df01ae0","Type":"ContainerStarted","Data":"bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1"}
Nov 03 19:23:04.196045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.195933    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:04.429047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.429017    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:23:04.429241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.429219    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" containerID="cri-o://ada854372cc8dd6b6a01a68d1480f6b8ae643cccff1ab94742a4309f669d54b9" gracePeriod=30
Nov 03 19:23:04.443317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.443285    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:04.448978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.448916    4944 scope.go:122] "RemoveContainer" containerID="b9fe4d76635c53ee515a41ae66334221ded356109e4ee85e7dbe40760fa8c03b"
Nov 03 19:23:04.568393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.568356    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44" event={"ID":"636a0f54-e9de-4d2d-b2c5-ab350df01ae0","Type":"ContainerStarted","Data":"ae76adf3bc354be8901d1853119420be935759f19690caa8a1e96b3392750813"}
Nov 03 19:23:04.677447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.677396    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44" podStartSLOduration=2.677381472 podStartE2EDuration="2.677381472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:04.676445341 +0000 UTC m=+2282.953798923" watchObservedRunningTime="2026-11-03 19:23:04.677381472 +0000 UTC m=+2282.954735054"
Nov 03 19:23:04.698563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.698528    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:04.990350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:04.990317    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:05.224439 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:23:05.224407    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:23:05.576941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:05.576902    4944 generic.go:356] "Generic (PLEG): container finished" podID="636a0f54-e9de-4d2d-b2c5-ab350df01ae0" containerID="ae76adf3bc354be8901d1853119420be935759f19690caa8a1e96b3392750813" exitCode=0
Nov 03 19:23:05.577108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:05.576988    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44" event={"ID":"636a0f54-e9de-4d2d-b2c5-ab350df01ae0","Type":"ContainerDied","Data":"ae76adf3bc354be8901d1853119420be935759f19690caa8a1e96b3392750813"}
Nov 03 19:23:05.954509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:05.954421    4944 patch_prober.go:29] interesting pod/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:23:05.954509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:05.954477    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:23:06.132298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.125744    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:06.270533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.270489    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"]
Nov 03 19:23:06.275117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.275072    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.368160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.367546    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7v4n\" (UniqueName: \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.368160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.367695    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.368160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.367781    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.370969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.370941    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"]
Nov 03 19:23:06.373722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.373695    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:06.468888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.468853    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7v4n\" (UniqueName: \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.469068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.468964    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.469068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.469036    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.469641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.469621    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.471338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.471308    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.529766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.529699    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7v4n\" (UniqueName: \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\") pod \"maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.606810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.599024    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:06.610844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.609486    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:23:06.656149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.656067    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.661407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.661341    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:23:06.709227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.709199    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-efe6c7\""
Nov 03 19:23:06.709411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.709199    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-efe6c7-dockercfg-574jb\""
Nov 03 19:23:06.734899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.734868    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:23:06.739717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.739202    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.777504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.777143    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbtl\" (UniqueName: \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.777504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.777275    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.777504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.777358    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prcrv\" (UniqueName: \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.777504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.777404    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.793956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.793781    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:23:06.853271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.853249    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:06.878313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.878276    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.878455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.878371    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbtl\" (UniqueName: \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.878525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.878495    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.878600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.878584    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prcrv\" (UniqueName: \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.879098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.879076    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.879291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.879272    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.935900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.935872    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prcrv\" (UniqueName: \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\") pod \"payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:06.936247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.936221    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbtl\" (UniqueName: \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\") pod \"payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") " pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:06.979053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.979023    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\" (UniqueName: \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\") pod \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") "
Nov 03 19:23:06.979194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.979076    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\") pod \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") "
Nov 03 19:23:06.979194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.979114    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\") pod \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\" (UID: \"636a0f54-e9de-4d2d-b2c5-ab350df01ae0\") "
Nov 03 19:23:06.979591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.979558    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca" pod "636a0f54-e9de-4d2d-b2c5-ab350df01ae0" (UID: "636a0f54-e9de-4d2d-b2c5-ab350df01ae0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:06.981062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.981038    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd" pod "636a0f54-e9de-4d2d-b2c5-ab350df01ae0" (UID: "636a0f54-e9de-4d2d-b2c5-ab350df01ae0"). InnerVolumeSpecName "kube-api-access-5qbdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:06.981155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.981070    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca" pod "636a0f54-e9de-4d2d-b2c5-ab350df01ae0" (UID: "636a0f54-e9de-4d2d-b2c5-ab350df01ae0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:23:06.988240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:06.988213    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:07.020526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.020498    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"]
Nov 03 19:23:07.025806 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:23:07.025764    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb955c15_e269_42ea_a281_72f673efd002.slice/crio-220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f WatchSource:0}: Error finding container 220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f: Status 404 returned error can't find the container with id 220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f
Nov 03 19:23:07.060350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.060132    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:07.080952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.080923    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:07.080952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.080953    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qbdd\" (UniqueName: \"kubernetes.io/projected/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-kube-api-access-5qbdd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:07.081100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.080969    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636a0f54-e9de-4d2d-b2c5-ab350df01ae0-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:07.368865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.368828    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:23:07.374251 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:23:07.374219    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826aa391_154d_40a3_9d0e_e5840556f19e.slice/crio-cd53d7e8f936c944d2b6b84b48097113c29d657519c55b3590a6108e01580674 WatchSource:0}: Error finding container cd53d7e8f936c944d2b6b84b48097113c29d657519c55b3590a6108e01580674: Status 404 returned error can't find the container with id cd53d7e8f936c944d2b6b84b48097113c29d657519c55b3590a6108e01580674
Nov 03 19:23:07.442439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.441601    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:23:07.447403 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:23:07.447372    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf27e9f8_9cef_482b_b3f7_a5f6570252cb.slice/crio-3732f8a1787167b7192b945e350aa4263c4c89fb46e9420fde2742fc33e9b64c WatchSource:0}: Error finding container 3732f8a1787167b7192b945e350aa4263c4c89fb46e9420fde2742fc33e9b64c: Status 404 returned error can't find the container with id 3732f8a1787167b7192b945e350aa4263c4c89fb46e9420fde2742fc33e9b64c
Nov 03 19:23:07.656000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.655904    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" event={"ID":"bf27e9f8-9cef-482b-b3f7-a5f6570252cb","Type":"ContainerStarted","Data":"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"}
Nov 03 19:23:07.656000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.655976    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:07.656227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.656007    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" event={"ID":"bf27e9f8-9cef-482b-b3f7-a5f6570252cb","Type":"ContainerStarted","Data":"3732f8a1787167b7192b945e350aa4263c4c89fb46e9420fde2742fc33e9b64c"}
Nov 03 19:23:07.657580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.657551    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" event={"ID":"826aa391-154d-40a3-9d0e-e5840556f19e","Type":"ContainerStarted","Data":"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"}
Nov 03 19:23:07.657732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.657602    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" event={"ID":"826aa391-154d-40a3-9d0e-e5840556f19e","Type":"ContainerStarted","Data":"cd53d7e8f936c944d2b6b84b48097113c29d657519c55b3590a6108e01580674"}
Nov 03 19:23:07.657732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.657629    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:07.659300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.659276    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l" event={"ID":"fb955c15-e269-42ea-a281-72f673efd002","Type":"ContainerStarted","Data":"ebcd7f507f91249e8456cf4b40716dd044d1cf4694a4c77cae4a1aa56e36a7d9"}
Nov 03 19:23:07.659610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.659329    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l" event={"ID":"fb955c15-e269-42ea-a281-72f673efd002","Type":"ContainerStarted","Data":"220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f"}
Nov 03 19:23:07.660735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.660714    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"
Nov 03 19:23:07.660862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.660750    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44" event={"ID":"636a0f54-e9de-4d2d-b2c5-ab350df01ae0","Type":"ContainerDied","Data":"bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1"}
Nov 03 19:23:07.660862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.660822    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd7faf8940c935549100c98bc4efae10143d943e5d8145c7566303223381eca1"
Nov 03 19:23:07.764381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.764327    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" podStartSLOduration=1.764313575 podStartE2EDuration="1.764313575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:07.762993053 +0000 UTC m=+2286.040346636" watchObservedRunningTime="2026-11-03 19:23:07.764313575 +0000 UTC m=+2286.041667158"
Nov 03 19:23:07.871409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:07.871351    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" podStartSLOduration=1.871332263 podStartE2EDuration="1.871332263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:07.870895281 +0000 UTC m=+2286.148248875" watchObservedRunningTime="2026-11-03 19:23:07.871332263 +0000 UTC m=+2286.148685845"
Nov 03 19:23:08.028441 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:08.028374    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l" podStartSLOduration=2.028353518 podStartE2EDuration="2.028353518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:23:08.026733818 +0000 UTC m=+2286.304087402" watchObservedRunningTime="2026-11-03 19:23:08.028353518 +0000 UTC m=+2286.305707102"
Nov 03 19:23:08.047807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:08.047762    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:08.668088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:08.668056    4944 generic.go:356] "Generic (PLEG): container finished" podID="fb955c15-e269-42ea-a281-72f673efd002" containerID="ebcd7f507f91249e8456cf4b40716dd044d1cf4694a4c77cae4a1aa56e36a7d9" exitCode=0
Nov 03 19:23:08.668555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:08.668139    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l" event={"ID":"fb955c15-e269-42ea-a281-72f673efd002","Type":"ContainerDied","Data":"ebcd7f507f91249e8456cf4b40716dd044d1cf4694a4c77cae4a1aa56e36a7d9"}
Nov 03 19:23:09.677043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.677012    4944 generic.go:356] "Generic (PLEG): container finished" podID="8d530c57-094d-4782-86d5-db5f87d9087f" containerID="ada854372cc8dd6b6a01a68d1480f6b8ae643cccff1ab94742a4309f669d54b9" exitCode=0
Nov 03 19:23:09.677463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.677082    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerDied","Data":"ada854372cc8dd6b6a01a68d1480f6b8ae643cccff1ab94742a4309f669d54b9"}
Nov 03 19:23:09.677463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.677146    4944 scope.go:122] "RemoveContainer" containerID="19adb32c8dcc29686b815ff94f027dc5b345460b097f29ac5b89930b18687f99"
Nov 03 19:23:09.724066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.724042    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:23:09.815841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.815810    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.815984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.815888    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.815984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.815945    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816038    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816075    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816145    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816191    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816224    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816258    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:09.816399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816290    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:09.816399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816309    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\") pod \"8d530c57-094d-4782-86d5-db5f87d9087f\" (UID: \"8d530c57-094d-4782-86d5-db5f87d9087f\") "
Nov 03 19:23:09.816574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816470    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:09.816574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816540    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:09.817982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816905    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:09.817982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816914    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.817982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816956    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.817982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816977    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.817982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.816991    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.818458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.818429    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:09.818621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.818601    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:23:09.818684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.818653    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:23:09.818740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.818715    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts" pod "8d530c57-094d-4782-86d5-db5f87d9087f" (UID: "8d530c57-094d-4782-86d5-db5f87d9087f"). InnerVolumeSpecName "kube-api-access-6bcts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:09.825216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.825198    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:09.917564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.917536    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\") pod \"fb955c15-e269-42ea-a281-72f673efd002\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") "
Nov 03 19:23:09.917769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.917750    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\") pod \"fb955c15-e269-42ea-a281-72f673efd002\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") "
Nov 03 19:23:09.917875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.917857    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\" (UniqueName: \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\") pod \"fb955c15-e269-42ea-a281-72f673efd002\" (UID: \"fb955c15-e269-42ea-a281-72f673efd002\") "
Nov 03 19:23:09.917998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.917979    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca" pod "fb955c15-e269-42ea-a281-72f673efd002" (UID: "fb955c15-e269-42ea-a281-72f673efd002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:09.918322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918301    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bcts\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-kube-api-access-6bcts\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.918322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918322    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d530c57-094d-4782-86d5-db5f87d9087f-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.918479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918331    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d530c57-094d-4782-86d5-db5f87d9087f-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.918479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918339    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d530c57-094d-4782-86d5-db5f87d9087f-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.918479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918348    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d530c57-094d-4782-86d5-db5f87d9087f-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.918479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.918357    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb955c15-e269-42ea-a281-72f673efd002-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:09.919824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.919783    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca" pod "fb955c15-e269-42ea-a281-72f673efd002" (UID: "fb955c15-e269-42ea-a281-72f673efd002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:23:09.919961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:09.919944    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n" pod "fb955c15-e269-42ea-a281-72f673efd002" (UID: "fb955c15-e269-42ea-a281-72f673efd002"). InnerVolumeSpecName "kube-api-access-w7v4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:10.019806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.019750    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb955c15-e269-42ea-a281-72f673efd002-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:10.019806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.019801    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7v4n\" (UniqueName: \"kubernetes.io/projected/fb955c15-e269-42ea-a281-72f673efd002-kube-api-access-w7v4n\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:10.056757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.056700    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:10.684803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.684769    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"
Nov 03 19:23:10.685212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.684821    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l" event={"ID":"fb955c15-e269-42ea-a281-72f673efd002","Type":"ContainerDied","Data":"220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f"}
Nov 03 19:23:10.685212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.684873    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="220155e4aab7105a790fa4e7dc2268330e9cd9c01316696e6c7b6423dea4da0f"
Nov 03 19:23:10.686482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.686457    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd" event={"ID":"8d530c57-094d-4782-86d5-db5f87d9087f","Type":"ContainerDied","Data":"11060c50fe303cc0d5a572a11feb50c2f97b95f2e4c229922e68bb63e8b0d8d2"}
Nov 03 19:23:10.686482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.686477    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"
Nov 03 19:23:10.686706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.686501    4944 scope.go:122] "RemoveContainer" containerID="ada854372cc8dd6b6a01a68d1480f6b8ae643cccff1ab94742a4309f669d54b9"
Nov 03 19:23:10.936695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.936664    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:23:10.982384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:10.982301    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-81a044-openshift-default-578f58f6df-ch6pd"]
Nov 03 19:23:11.936778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:11.936745    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" path="/var/lib/kubelet/pods/8d530c57-094d-4782-86d5-db5f87d9087f/volumes"
Nov 03 19:23:12.901645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:12.901610    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:13.836816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:13.836755    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:14.089004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:14.088917    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:17.936969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:17.936940    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 19:23:17.940600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:17.940560    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:18.035432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:18.035401    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 19:23:18.385145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:18.385089    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:18.631182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:18.631148    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:18.671104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:18.671030    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:23:18.671273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:18.671257    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:23:19.359813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:19.358843    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:19.615724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:19.615630    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:22.894084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:22.894041    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:22.937731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:22.937693    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing" containerID="cri-o://544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed" gracePeriod=25
Nov 03 19:23:23.036521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:23.036473    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing" containerID="cri-o://1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd" gracePeriod=25
Nov 03 19:23:27.656313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:27.656280    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:30.097521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:30.097487    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:31.078889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:31.078847    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:23:31.078889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:31.078868    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:23:31.596236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:31.596201    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:33.117105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.117068    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:33.363203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.363169    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:33.489403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.489350    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"]
Nov 03 19:23:33.534516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.534486    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1cc3f2-mxwxj"]
Nov 03 19:23:33.606645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.606614    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:33.936043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.935966    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e91da8-ec07-48a9-89fb-9d2cbfdb260f" path="/var/lib/kubelet/pods/18e91da8-ec07-48a9-89fb-9d2cbfdb260f/volumes"
Nov 03 19:23:33.989903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:33.989867    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:35.224483 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:23:35.224448    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:23:38.166447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:38.166404    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:39.863248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:39.863208    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:40.055429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.055397    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"]
Nov 03 19:23:40.069394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.069364    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc9d86a4-qpbss"]
Nov 03 19:23:40.112629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.112595    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:40.357982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.357948    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:40.731749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.731670    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:40.893617 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.893583    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"]
Nov 03 19:23:40.894542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894518    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fb955c15-e269-42ea-a281-72f673efd002" containerName="revoke-keys"
Nov 03 19:23:40.894542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894542    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb955c15-e269-42ea-a281-72f673efd002" containerName="revoke-keys"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894570    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894579    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894610    4944 cpu_manager.go:414] "containerMap: removing container" podUID="636a0f54-e9de-4d2d-b2c5-ab350df01ae0" containerName="revoke-keys"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894619    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636a0f54-e9de-4d2d-b2c5-ab350df01ae0" containerName="revoke-keys"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894645    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:23:40.894744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894652    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:23:40.895158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894836    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb955c15-e269-42ea-a281-72f673efd002" containerName="revoke-keys"
Nov 03 19:23:40.895158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894855    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:23:40.895158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.894870    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="636a0f54-e9de-4d2d-b2c5-ab350df01ae0" containerName="revoke-keys"
Nov 03 19:23:40.900020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.899999    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:40.948932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.948900    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:23:40.952115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.952080    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"]
Nov 03 19:23:40.994250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.994193    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:23:40.994250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.994193    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:23:40.994450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:40.994299    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:23:41.022140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.022109    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs77z\" (UniqueName: \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.022297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.022274    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.022400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.022382    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.078250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.078216    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:23:41.078250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.078227    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:23:41.123305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.123274    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.123457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.123406    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs77z\" (UniqueName: \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.123584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.123559    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.124013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.123993    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.125869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.125847    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.180844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.180817    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs77z\" (UniqueName: \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\") pod \"maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.214025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.214001    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:41.587270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.587186    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"]
Nov 03 19:23:41.593713 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:23:41.593684    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f58d04_97e0_4ce5_a83f_327c5548fa2a.slice/crio-864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f WatchSource:0}: Error finding container 864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f: Status 404 returned error can't find the container with id 864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f
Nov 03 19:23:41.905540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.905452    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk" event={"ID":"e4f58d04-97e0-4ce5-a83f-327c5548fa2a","Type":"ContainerStarted","Data":"864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f"}
Nov 03 19:23:41.936447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:41.936419    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148d862c-34fc-4633-9ea2-d5d690b36bd9" path="/var/lib/kubelet/pods/148d862c-34fc-4633-9ea2-d5d690b36bd9/volumes"
Nov 03 19:23:42.352095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:42.352065    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:42.913339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:42.913304    4944 generic.go:356] "Generic (PLEG): container finished" podID="e4f58d04-97e0-4ce5-a83f-327c5548fa2a" containerID="172b0d9e8626b7a28bfece447e2bc9e8b7c53b2bd72efa442cfc7edfc284938a" exitCode=0
Nov 03 19:23:42.913726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:42.913393    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk" event={"ID":"e4f58d04-97e0-4ce5-a83f-327c5548fa2a","Type":"ContainerDied","Data":"172b0d9e8626b7a28bfece447e2bc9e8b7c53b2bd72efa442cfc7edfc284938a"}
Nov 03 19:23:43.216691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.212591    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:43.261472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.261447    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 19:23:43.361210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.361184    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\") pod \"7f0f517d-2724-4285-b993-edfee82810d0\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") "
Nov 03 19:23:43.361354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.361225    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\" (UniqueName: \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\") pod \"7f0f517d-2724-4285-b993-edfee82810d0\" (UID: \"7f0f517d-2724-4285-b993-edfee82810d0\") "
Nov 03 19:23:43.361648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.361620    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume" pod "7f0f517d-2724-4285-b993-edfee82810d0" (UID: "7f0f517d-2724-4285-b993-edfee82810d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:43.361881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.361860    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f0f517d-2724-4285-b993-edfee82810d0-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:43.363290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.363269    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46" pod "7f0f517d-2724-4285-b993-edfee82810d0" (UID: "7f0f517d-2724-4285-b993-edfee82810d0"). InnerVolumeSpecName "kube-api-access-qqw46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:43.389844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.389820    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 19:23:43.463536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.463507    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqw46\" (UniqueName: \"kubernetes.io/projected/7f0f517d-2724-4285-b993-edfee82810d0-kube-api-access-qqw46\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:43.564039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.564007    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\" (UniqueName: \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\") pod \"8592e418-c7a0-4c21-b5fb-719097dc10f5\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") "
Nov 03 19:23:43.564227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.564094    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\") pod \"8592e418-c7a0-4c21-b5fb-719097dc10f5\" (UID: \"8592e418-c7a0-4c21-b5fb-719097dc10f5\") "
Nov 03 19:23:43.564483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.564459    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume" pod "8592e418-c7a0-4c21-b5fb-719097dc10f5" (UID: "8592e418-c7a0-4c21-b5fb-719097dc10f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:43.566176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.566149    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp" pod "8592e418-c7a0-4c21-b5fb-719097dc10f5" (UID: "8592e418-c7a0-4c21-b5fb-719097dc10f5"). InnerVolumeSpecName "kube-api-access-cj4wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:43.666317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.666286    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj4wp\" (UniqueName: \"kubernetes.io/projected/8592e418-c7a0-4c21-b5fb-719097dc10f5-kube-api-access-cj4wp\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:43.666317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.666313    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8592e418-c7a0-4c21-b5fb-719097dc10f5-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:43.922488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.922391    4944 generic.go:356] "Generic (PLEG): container finished" podID="7f0f517d-2724-4285-b993-edfee82810d0" containerID="544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed" exitCode=0
Nov 03 19:23:43.922488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.922437    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"
Nov 03 19:23:43.922488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.922471    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" event={"ID":"7f0f517d-2724-4285-b993-edfee82810d0","Type":"ContainerDied","Data":"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"}
Nov 03 19:23:43.923211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.922565    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7" event={"ID":"7f0f517d-2724-4285-b993-edfee82810d0","Type":"ContainerDied","Data":"4d463a7e0079c8d38cc5b2c6bc5fbe2fed57678eb78fcdf76408cbd94ce4da6e"}
Nov 03 19:23:43.923211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.922592    4944 scope.go:122] "RemoveContainer" containerID="544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"
Nov 03 19:23:43.924445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.924420    4944 generic.go:356] "Generic (PLEG): container finished" podID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerID="1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd" exitCode=0
Nov 03 19:23:43.924572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.924472    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" event={"ID":"8592e418-c7a0-4c21-b5fb-719097dc10f5","Type":"ContainerDied","Data":"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"}
Nov 03 19:23:43.924572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.924499    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"
Nov 03 19:23:43.924572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.924526    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j" event={"ID":"8592e418-c7a0-4c21-b5fb-719097dc10f5","Type":"ContainerDied","Data":"3cb0dcff78883c19d4990e4b90fdc8c56521ca572debbf42e818f35c8fe12cca"}
Nov 03 19:23:43.936452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.936432    4944 scope.go:122] "RemoveContainer" containerID="544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"
Nov 03 19:23:43.937088 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:23:43.937055    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed\": container with ID starting with 544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed not found: ID does not exist" containerID="544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"
Nov 03 19:23:43.937208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.937099    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed"} err="failed to get container status \"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed\": rpc error: code = NotFound desc = could not find container \"544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed\": container with ID starting with 544ad4c12f4a266679af8910af7b129aa4b9e56de857e48097a92cffc3c63fed not found: ID does not exist"
Nov 03 19:23:43.937208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.937123    4944 scope.go:122] "RemoveContainer" containerID="1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"
Nov 03 19:23:43.948010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.947991    4944 scope.go:122] "RemoveContainer" containerID="1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"
Nov 03 19:23:43.948297 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:23:43.948279    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd\": container with ID starting with 1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd not found: ID does not exist" containerID="1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"
Nov 03 19:23:43.948361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:43.948303    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd"} err="failed to get container status \"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd\": rpc error: code = NotFound desc = could not find container \"1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd\": container with ID starting with 1834303c6210d4d799761aa65d7d79757c83ac69cc4406177ff7e895417302bd not found: ID does not exist"
Nov 03 19:23:44.062504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.062477    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:44.077145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.077107    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 19:23:44.083196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.083171    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98860e-6b8768764c-gjc8j"]
Nov 03 19:23:44.171610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.171577    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\") pod \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") "
Nov 03 19:23:44.171847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.171746    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\" (UniqueName: \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\") pod \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") "
Nov 03 19:23:44.171847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.171828    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\") pod \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\" (UID: \"e4f58d04-97e0-4ce5-a83f-327c5548fa2a\") "
Nov 03 19:23:44.172066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.172029    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca" pod "e4f58d04-97e0-4ce5-a83f-327c5548fa2a" (UID: "e4f58d04-97e0-4ce5-a83f-327c5548fa2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:23:44.172458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.172434    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:44.174332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.174271    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca" pod "e4f58d04-97e0-4ce5-a83f-327c5548fa2a" (UID: "e4f58d04-97e0-4ce5-a83f-327c5548fa2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:23:44.174652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.174577    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z" pod "e4f58d04-97e0-4ce5-a83f-327c5548fa2a" (UID: "e4f58d04-97e0-4ce5-a83f-327c5548fa2a"). InnerVolumeSpecName "kube-api-access-hs77z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:23:44.186449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.186425    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 19:23:44.191553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.191533    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-98860e-9d6bdb87d-xw9s7"]
Nov 03 19:23:44.273414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.273377    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs77z\" (UniqueName: \"kubernetes.io/projected/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-kube-api-access-hs77z\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:44.273414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.273411    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4f58d04-97e0-4ce5-a83f-327c5548fa2a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:23:44.934213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.934187    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"
Nov 03 19:23:44.934713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.934217    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk" event={"ID":"e4f58d04-97e0-4ce5-a83f-327c5548fa2a","Type":"ContainerDied","Data":"864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f"}
Nov 03 19:23:44.934713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:44.934261    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864cfd21c355a3f3e7265515a32b89a68bcaeee488f6a9491f205b3f4e397e8f"
Nov 03 19:23:45.584212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:45.584180    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:23:45.854377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:45.854293    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:45.937841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:45.937806    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0f517d-2724-4285-b993-edfee82810d0" path="/var/lib/kubelet/pods/7f0f517d-2724-4285-b993-edfee82810d0/volumes"
Nov 03 19:23:45.938501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:45.938480    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" path="/var/lib/kubelet/pods/8592e418-c7a0-4c21-b5fb-719097dc10f5/volumes"
Nov 03 19:23:46.933067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:46.933030    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:47.180619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:47.180590    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:47.774881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:47.774848    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:48.540349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:48.540319    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:52.377039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:52.376997    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:53.435249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:53.435217    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:54.911226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:54.911186    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 19:23:55.009818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:55.009757    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 19:23:55.230883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:55.230779    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:57.183219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:57.183185    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:57.460934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:57.460845    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:23:59.911521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:23:59.911477    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing" containerID="cri-o://3fba2c53ad484f1db241d621c6407e96927be5b32d936b44e43d1e62018a4237" gracePeriod=25
Nov 03 19:24:00.010636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:00.010567    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing" containerID="cri-o://73fef8062f85892a53f6af6dae0f6ef4fd682cd87f61d838623051721c649b97" gracePeriod=25
Nov 03 19:24:00.606588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:00.606540    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main" containerID="cri-o://f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb" gracePeriod=45
Nov 03 19:24:00.879053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:00.879030    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:24:01.005081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005046    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\" (UniqueName: \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.005578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005146    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.005578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005258    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.005578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005347    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.005578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005417    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.005578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.005492    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\") pod \"a8e7c020-472d-4131-806d-8ea0847e3a3d\" (UID: \"a8e7c020-472d-4131-806d-8ea0847e3a3d\") "
Nov 03 19:24:01.006284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.006063    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:01.006284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.006084    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:01.006284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.006103    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:01.008358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.008329    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:24:01.008935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.008911    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "kube-api-access-57scn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:01.009165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.009146    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs" pod "a8e7c020-472d-4131-806d-8ea0847e3a3d" (UID: "a8e7c020-472d-4131-806d-8ea0847e3a3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:01.047097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.047060    4944 generic.go:356] "Generic (PLEG): container finished" podID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerID="f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb" exitCode=0
Nov 03 19:24:01.047279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.047116    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"
Nov 03 19:24:01.047279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.047157    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" event={"ID":"a8e7c020-472d-4131-806d-8ea0847e3a3d","Type":"ContainerDied","Data":"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"}
Nov 03 19:24:01.047279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.047206    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5" event={"ID":"a8e7c020-472d-4131-806d-8ea0847e3a3d","Type":"ContainerDied","Data":"0ae006a3e9cde63fd33d5e954b7e7aa971648f060ec96a6b1c144d0ba9d11260"}
Nov 03 19:24:01.047279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.047232    4944 scope.go:122] "RemoveContainer" containerID="f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"
Nov 03 19:24:01.060482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.060460    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:01.067028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.067001    4944 scope.go:122] "RemoveContainer" containerID="f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"
Nov 03 19:24:01.067401 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:24:01.067362    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb\": container with ID starting with f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb not found: ID does not exist" containerID="f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"
Nov 03 19:24:01.067512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.067414    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb"} err="failed to get container status \"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb\": rpc error: code = NotFound desc = could not find container \"f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb\": container with ID starting with f130da4e054d68a0ca2f80a1f07b1319cb58e9e26cbdf891dc58c084af122cdb not found: ID does not exist"
Nov 03 19:24:01.106914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106886    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57scn\" (UniqueName: \"kubernetes.io/projected/a8e7c020-472d-4131-806d-8ea0847e3a3d-kube-api-access-57scn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.106914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106913    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.107138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106926    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8e7c020-472d-4131-806d-8ea0847e3a3d-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.107138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106934    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.107138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106943    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.107138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.106951    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8e7c020-472d-4131-806d-8ea0847e3a3d-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:01.204622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.204516    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:24:01.210049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.210022    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-90a9bf/distinct-2-w5-90a9bf-kserve-bf4fd6fd4-x96s5"]
Nov 03 19:24:01.346178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.327638    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:01.937527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:01.937493    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" path="/var/lib/kubelet/pods/a8e7c020-472d-4131-806d-8ea0847e3a3d/volumes"
Nov 03 19:24:02.376589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:02.376548    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:03.785774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:03.785744    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:04.570840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:04.570809    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:04.823295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:04.823208    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:05.224581 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:24:05.224498    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:24:06.290104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:06.290064    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:24:06.290518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:06.290075    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:24:06.631764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:06.631665    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:07.789529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:07.789487    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:08.042890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.042767    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:08.292184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.292148    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:08.370592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.370411    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:24:08.371748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371715    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e4f58d04-97e0-4ce5-a83f-327c5548fa2a" containerName="revoke-keys"
Nov 03 19:24:08.371748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371740    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f58d04-97e0-4ce5-a83f-327c5548fa2a" containerName="revoke-keys"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371773    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371781    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371808    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371816    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371855    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing"
Nov 03 19:24:08.371972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.371862    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing"
Nov 03 19:24:08.372274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.372003    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8592e418-c7a0-4c21-b5fb-719097dc10f5" containerName="payload-pre-processing"
Nov 03 19:24:08.372274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.372023    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d530c57-094d-4782-86d5-db5f87d9087f" containerName="istio-proxy"
Nov 03 19:24:08.372274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.372034    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f58d04-97e0-4ce5-a83f-327c5548fa2a" containerName="revoke-keys"
Nov 03 19:24:08.372274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.372045    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e7c020-472d-4131-806d-8ea0847e3a3d" containerName="main"
Nov 03 19:24:08.372274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.372056    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0f517d-2724-4285-b993-edfee82810d0" containerName="payload-processing"
Nov 03 19:24:08.377691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.377667    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.422734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.422702    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:24:08.464556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.464520    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c3e530\""
Nov 03 19:24:08.464556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.464535    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c3e530-dockercfg-j7679\""
Nov 03 19:24:08.512215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.512178    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmx2\" (UniqueName: \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.512386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.512306    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.555756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.555722    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:08.617643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.617599    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.617863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.617821    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmx2\" (UniqueName: \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.620563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.620534    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.675191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.675122    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmx2\" (UniqueName: \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\") pod \"payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.703112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.703081    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:08.811847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.811753    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:08.871073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.871043    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"]
Nov 03 19:24:08.875847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:08.875825    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd1327-9rhgh"]
Nov 03 19:24:09.140147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:09.140116    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:24:09.145683 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:09.145659    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf305f266_bb0c_473b_92a1_ebdb03c2f9d7.slice/crio-3fdcf3d17af7edd29ed12eca5049e709ea5b3b56d3455ec01175cc5fd9f4a0b7 WatchSource:0}: Error finding container 3fdcf3d17af7edd29ed12eca5049e709ea5b3b56d3455ec01175cc5fd9f4a0b7: Status 404 returned error can't find the container with id 3fdcf3d17af7edd29ed12eca5049e709ea5b3b56d3455ec01175cc5fd9f4a0b7
Nov 03 19:24:09.422948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:09.422859    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:09.739677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:09.739649    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:09.934461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:09.934426    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e324dc9c-05f9-4ba4-9738-bdea05154325" path="/var/lib/kubelet/pods/e324dc9c-05f9-4ba4-9738-bdea05154325/volumes"
Nov 03 19:24:10.110891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.110763    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" event={"ID":"f305f266-bb0c-473b-92a1-ebdb03c2f9d7","Type":"ContainerStarted","Data":"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"}
Nov 03 19:24:10.110891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.110846    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:10.110891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.110871    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" event={"ID":"f305f266-bb0c-473b-92a1-ebdb03c2f9d7","Type":"ContainerStarted","Data":"3fdcf3d17af7edd29ed12eca5049e709ea5b3b56d3455ec01175cc5fd9f4a0b7"}
Nov 03 19:24:10.220969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.220917    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" podStartSLOduration=2.22090248 podStartE2EDuration="2.22090248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:10.21853467 +0000 UTC m=+2348.495888266" watchObservedRunningTime="2026-11-03 19:24:10.22090248 +0000 UTC m=+2348.498256064"
Nov 03 19:24:10.237851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.237820    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:10.507268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.507230    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:10.613190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.613155    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"]
Nov 03 19:24:10.615933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:10.615903    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d86d1cd0-h6h44"]
Nov 03 19:24:11.225993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:11.225959    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:11.935176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:11.935147    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636a0f54-e9de-4d2d-b2c5-ab350df01ae0" path="/var/lib/kubelet/pods/636a0f54-e9de-4d2d-b2c5-ab350df01ae0/volumes"
Nov 03 19:24:14.707574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:14.707536    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:14.954150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:14.954117    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:15.209093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:15.209059    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:15.676411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:15.676379    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:15.937568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:15.937478    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:16.181074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:16.181014    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:16.289957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:16.289923    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:24:16.289957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:16.289947    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:24:16.425814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:16.425773    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:16.864600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:16.864569    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:17.498763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:17.498723    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:17.851639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:17.851551    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:19.798937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:19.798904    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:20.188458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.188424    4944 generic.go:356] "Generic (PLEG): container finished" podID="c368dcb7-83e7-45ef-8202-6072c0250343" containerID="73fef8062f85892a53f6af6dae0f6ef4fd682cd87f61d838623051721c649b97" exitCode=0
Nov 03 19:24:20.188609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.188492    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" event={"ID":"c368dcb7-83e7-45ef-8202-6072c0250343","Type":"ContainerDied","Data":"73fef8062f85892a53f6af6dae0f6ef4fd682cd87f61d838623051721c649b97"}
Nov 03 19:24:20.191083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.190220    4944 generic.go:356] "Generic (PLEG): container finished" podID="287445df-2f48-41d0-b80a-1149b7619398" containerID="3fba2c53ad484f1db241d621c6407e96927be5b32d936b44e43d1e62018a4237" exitCode=0
Nov 03 19:24:20.191083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.190282    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" event={"ID":"287445df-2f48-41d0-b80a-1149b7619398","Type":"ContainerDied","Data":"3fba2c53ad484f1db241d621c6407e96927be5b32d936b44e43d1e62018a4237"}
Nov 03 19:24:20.219204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.219170    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 19:24:20.263047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.263006    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\" (UniqueName: \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\") pod \"287445df-2f48-41d0-b80a-1149b7619398\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") "
Nov 03 19:24:20.263214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.263196    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\") pod \"287445df-2f48-41d0-b80a-1149b7619398\" (UID: \"287445df-2f48-41d0-b80a-1149b7619398\") "
Nov 03 19:24:20.263666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.263634    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume" pod "287445df-2f48-41d0-b80a-1149b7619398" (UID: "287445df-2f48-41d0-b80a-1149b7619398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:20.265265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.265237    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g" pod "287445df-2f48-41d0-b80a-1149b7619398" (UID: "287445df-2f48-41d0-b80a-1149b7619398"). InnerVolumeSpecName "kube-api-access-jsd9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:20.286341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.286320    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 19:24:20.369321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.369231    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\" (UniqueName: \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\") pod \"c368dcb7-83e7-45ef-8202-6072c0250343\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") "
Nov 03 19:24:20.369491 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.369326    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\") pod \"c368dcb7-83e7-45ef-8202-6072c0250343\" (UID: \"c368dcb7-83e7-45ef-8202-6072c0250343\") "
Nov 03 19:24:20.369704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.369689    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/287445df-2f48-41d0-b80a-1149b7619398-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:20.369768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.369706    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsd9g\" (UniqueName: \"kubernetes.io/projected/287445df-2f48-41d0-b80a-1149b7619398-kube-api-access-jsd9g\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:20.370036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.370017    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume" pod "c368dcb7-83e7-45ef-8202-6072c0250343" (UID: "c368dcb7-83e7-45ef-8202-6072c0250343"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:20.372076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.372048    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc" pod "c368dcb7-83e7-45ef-8202-6072c0250343" (UID: "c368dcb7-83e7-45ef-8202-6072c0250343"). InnerVolumeSpecName "kube-api-access-6bvdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:20.470744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.470709    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvdc\" (UniqueName: \"kubernetes.io/projected/c368dcb7-83e7-45ef-8202-6072c0250343-kube-api-access-6bvdc\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:20.470744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:20.470741    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c368dcb7-83e7-45ef-8202-6072c0250343-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:21.119843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.119815    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:24:21.203723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.203626    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns" event={"ID":"287445df-2f48-41d0-b80a-1149b7619398","Type":"ContainerDied","Data":"c9fe23a6e96a413d7e19e85759d1bbfc4ef556b6f0e99fcbf722e419613f4478"}
Nov 03 19:24:21.203723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.203695    4944 scope.go:122] "RemoveContainer" containerID="3fba2c53ad484f1db241d621c6407e96927be5b32d936b44e43d1e62018a4237"
Nov 03 19:24:21.204016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.203858    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"
Nov 03 19:24:21.217234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.217200    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7" event={"ID":"c368dcb7-83e7-45ef-8202-6072c0250343","Type":"ContainerDied","Data":"9f5d41409a1b8f8c137e7bd2af52747e63db355b2d5e4f94e607867c41d38be1"}
Nov 03 19:24:21.217618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.217533    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"
Nov 03 19:24:21.276281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.276248    4944 scope.go:122] "RemoveContainer" containerID="73fef8062f85892a53f6af6dae0f6ef4fd682cd87f61d838623051721c649b97"
Nov 03 19:24:21.314890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.314848    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:21.436260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.436228    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 19:24:21.439954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.439925    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-90a9bf-c98df7fc4-2cqns"]
Nov 03 19:24:21.544907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.544875    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 19:24:21.551218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.551194    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-90a9bf-ffd978d87-npht7"]
Nov 03 19:24:21.578858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.578828    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:21.934374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.934299    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287445df-2f48-41d0-b80a-1149b7619398" path="/var/lib/kubelet/pods/287445df-2f48-41d0-b80a-1149b7619398/volumes"
Nov 03 19:24:21.934745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:21.934731    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" path="/var/lib/kubelet/pods/c368dcb7-83e7-45ef-8202-6072c0250343/volumes"
Nov 03 19:24:22.775044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:22.774838    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:23.027784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:23.027693    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:23.953973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:23.953942    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:24.068654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.068612    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"]
Nov 03 19:24:24.069270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069249    4944 cpu_manager.go:414] "containerMap: removing container" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing"
Nov 03 19:24:24.069387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069270    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing"
Nov 03 19:24:24.069387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069302    4944 cpu_manager.go:414] "containerMap: removing container" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing"
Nov 03 19:24:24.069387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069310    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing"
Nov 03 19:24:24.069561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069476    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="287445df-2f48-41d0-b80a-1149b7619398" containerName="payload-processing"
Nov 03 19:24:24.069561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.069494    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="c368dcb7-83e7-45ef-8202-6072c0250343" containerName="payload-pre-processing"
Nov 03 19:24:24.074556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.074532    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.115311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.115270    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.115465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.115425    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.115548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.115472    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqms\" (UniqueName: \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.121386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.121360    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"]
Nov 03 19:24:24.125983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.125961    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"]
Nov 03 19:24:24.126119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.126053    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.164409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.164383    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:24:24.164409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.164383    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:24:24.164577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.164445    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:24:24.169065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.169047    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:24:24.179015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.178986    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"]
Nov 03 19:24:24.216978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.216907    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhfg\" (UniqueName: \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.216978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.216948    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.217144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.217046    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.217182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.217146    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.217264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.217180    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqms\" (UniqueName: \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.217264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.217223    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.217688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.217669    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.219467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.219444    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.274323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.274293    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqms\" (UniqueName: \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\") pod \"maas-api-revoke-sub-b1fdf6e1-pddmr\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.282511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.282483    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"]
Nov 03 19:24:24.286819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.286768    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.318029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.317994    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhfg\" (UniqueName: \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.318151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318033    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7dq\" (UniqueName: \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.318151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318057    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.318260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318240    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.318298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318279    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.318340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318315    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.318890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.318869    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.320400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.320378    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.338281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.338256    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"]
Nov 03 19:24:24.342743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.342719    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"]
Nov 03 19:24:24.342878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.342851    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.375555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.375523    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhfg\" (UniqueName: \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\") pod \"maas-api-revoke-sub-bc8aec8c-7xxml\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.387516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.387497    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:24.394822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.394777    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"]
Nov 03 19:24:24.420226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420188    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.420390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420256    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.420390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420378    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7dq\" (UniqueName: \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.420533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420480    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.420591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420571    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hmh\" (UniqueName: \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.420673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.420655    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.421633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.421602    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.423478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.423452    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.441189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.441156    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:24.477559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.477486    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7dq\" (UniqueName: \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\") pod \"maas-api-revoke-sub-22f4af47-dv8zg\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.521547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.521514    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.522382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.521670    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.522382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.521755    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hmh\" (UniqueName: \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.522382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.522347    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.529315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.529291    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.578909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.578878    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hmh\" (UniqueName: \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\") pod \"maas-api-revoke-sub-54598209-ksrgj\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.600720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.600690    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:24.659175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.659147    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:24.762056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.762025    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"]
Nov 03 19:24:24.771970 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:24.771931    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode947238f_4bd7_4953_806f_c59544f1b53b.slice/crio-bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a WatchSource:0}: Error finding container bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a: Status 404 returned error can't find the container with id bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a
Nov 03 19:24:24.827401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.827369    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"]
Nov 03 19:24:24.832376 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:24.832350    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb866af48_68a2_449c_afe5_1e64e05619f4.slice/crio-5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869 WatchSource:0}: Error finding container 5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869: Status 404 returned error can't find the container with id 5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869
Nov 03 19:24:24.972093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:24.972062    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"]
Nov 03 19:24:25.009670 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:25.009637    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b07bdbe_61e3_429e_8b89_08fce8322677.slice/crio-bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7 WatchSource:0}: Error finding container bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7: Status 404 returned error can't find the container with id bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7
Nov 03 19:24:25.043291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.043267    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"]
Nov 03 19:24:25.048703 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:25.048679    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8616bd_110e_4f79_b251_b4a15802f826.slice/crio-cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065 WatchSource:0}: Error finding container cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065: Status 404 returned error can't find the container with id cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065
Nov 03 19:24:25.247027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.246992    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg" event={"ID":"2b07bdbe-61e3-429e-8b89-08fce8322677","Type":"ContainerStarted","Data":"bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7"}
Nov 03 19:24:25.248189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.248164    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj" event={"ID":"9c8616bd-110e-4f79-b251-b4a15802f826","Type":"ContainerStarted","Data":"cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065"}
Nov 03 19:24:25.249695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.249669    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr" event={"ID":"e947238f-4bd7-4953-806f-c59544f1b53b","Type":"ContainerStarted","Data":"c2ac2fc62c92235b990a56e459f1dda66a9cffa2b3d8a313b430ce3849a4ca52"}
Nov 03 19:24:25.249827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.249722    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr" event={"ID":"e947238f-4bd7-4953-806f-c59544f1b53b","Type":"ContainerStarted","Data":"bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a"}
Nov 03 19:24:25.251340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.251319    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml" event={"ID":"b866af48-68a2-449c-afe5-1e64e05619f4","Type":"ContainerStarted","Data":"62ca77cb7fd47dcbf8ad6ff9639e11f2edfe7b31f4e5f156b724f7f7223dc13e"}
Nov 03 19:24:25.251442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.251365    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml" event={"ID":"b866af48-68a2-449c-afe5-1e64e05619f4","Type":"ContainerStarted","Data":"5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869"}
Nov 03 19:24:25.404339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.404286    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr" podStartSLOduration=1.404272119 podStartE2EDuration="1.404272119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:25.401758118 +0000 UTC m=+2363.679111700" watchObservedRunningTime="2026-11-03 19:24:25.404272119 +0000 UTC m=+2363.681625701"
Nov 03 19:24:25.445844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.445777    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:25.516128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.516079    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml" podStartSLOduration=1.516061862 podStartE2EDuration="1.516061862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:25.51387282 +0000 UTC m=+2363.791226404" watchObservedRunningTime="2026-11-03 19:24:25.516061862 +0000 UTC m=+2363.793415445"
Nov 03 19:24:25.708633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.705996    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:25.955349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.955269    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:25.983915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.983881    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:24:25.987876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:25.987857    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.038400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.038367    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-73a216-dockercfg-j6879\""
Nov 03 19:24:26.038592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.038400    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-73a216\""
Nov 03 19:24:26.042674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.042650    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:24:26.042823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.042784    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skljt\" (UniqueName: \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.043044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.043023    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.144063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.144023    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skljt\" (UniqueName: \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.144277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.144172    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.145198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.145159    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.190090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.190052    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:24:26.194374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.194352    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.202154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.202132    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skljt\" (UniqueName: \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\") pod \"payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.241510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.241481    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:24:26.245854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.245821    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.246000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.245869    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t9g6\" (UniqueName: \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.259642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.259406    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg" event={"ID":"2b07bdbe-61e3-429e-8b89-08fce8322677","Type":"ContainerStarted","Data":"31ffc99db66f78d4ecfdd40dab57693c9f66e116930a05c0292080124ecd2890"}
Nov 03 19:24:26.260995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.260970    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj" event={"ID":"9c8616bd-110e-4f79-b251-b4a15802f826","Type":"ContainerStarted","Data":"cd81ee739f98cedd0a9d2ecb05909810e836565af873ed258ec3ac584a554c04"}
Nov 03 19:24:26.262510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.262487    4944 generic.go:356] "Generic (PLEG): container finished" podID="e947238f-4bd7-4953-806f-c59544f1b53b" containerID="c2ac2fc62c92235b990a56e459f1dda66a9cffa2b3d8a313b430ce3849a4ca52" exitCode=0
Nov 03 19:24:26.262601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.262570    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr" event={"ID":"e947238f-4bd7-4953-806f-c59544f1b53b","Type":"ContainerDied","Data":"c2ac2fc62c92235b990a56e459f1dda66a9cffa2b3d8a313b430ce3849a4ca52"}
Nov 03 19:24:26.264118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.264097    4944 generic.go:356] "Generic (PLEG): container finished" podID="b866af48-68a2-449c-afe5-1e64e05619f4" containerID="62ca77cb7fd47dcbf8ad6ff9639e11f2edfe7b31f4e5f156b724f7f7223dc13e" exitCode=0
Nov 03 19:24:26.264213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.264128    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml" event={"ID":"b866af48-68a2-449c-afe5-1e64e05619f4","Type":"ContainerDied","Data":"62ca77cb7fd47dcbf8ad6ff9639e11f2edfe7b31f4e5f156b724f7f7223dc13e"}
Nov 03 19:24:26.301047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.301017    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:26.346945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.346881    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.347117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.346964    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t9g6\" (UniqueName: \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.347666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.347640    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.404621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.404588    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t9g6\" (UniqueName: \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\") pod \"payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.517540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.517352    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg" podStartSLOduration=2.517333477 podStartE2EDuration="2.517333477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:26.517153147 +0000 UTC m=+2364.794506730" watchObservedRunningTime="2026-11-03 19:24:26.517333477 +0000 UTC m=+2364.794687079"
Nov 03 19:24:26.522323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.522298    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:26.627064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.627015    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj" podStartSLOduration=2.6270011970000002 podStartE2EDuration="2.627001197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:26.624729715 +0000 UTC m=+2364.902083297" watchObservedRunningTime="2026-11-03 19:24:26.627001197 +0000 UTC m=+2364.904354779"
Nov 03 19:24:26.674613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.674580    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:24:26.679780 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:26.679751    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a35f35_56da_4a98_85c6_1efb41f2aadc.slice/crio-68abb4f1b6fb639c2b1402ba24c1d8a6401d27f928a1da47d031052cb4268b5b WatchSource:0}: Error finding container 68abb4f1b6fb639c2b1402ba24c1d8a6401d27f928a1da47d031052cb4268b5b: Status 404 returned error can't find the container with id 68abb4f1b6fb639c2b1402ba24c1d8a6401d27f928a1da47d031052cb4268b5b
Nov 03 19:24:26.896585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:26.896503    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:24:26.901614 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:24:26.901587    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f3a3d0_430d_4591_b943_e2231afb753e.slice/crio-955f4eb36756caa9d5e65cd90ae42dad0ac95d641af314decdc6cfa11d1e65aa WatchSource:0}: Error finding container 955f4eb36756caa9d5e65cd90ae42dad0ac95d641af314decdc6cfa11d1e65aa: Status 404 returned error can't find the container with id 955f4eb36756caa9d5e65cd90ae42dad0ac95d641af314decdc6cfa11d1e65aa
Nov 03 19:24:27.273192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.273150    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" event={"ID":"77f3a3d0-430d-4591-b943-e2231afb753e","Type":"ContainerStarted","Data":"572266df77b2eebb0057f5a69776c313250daa5861b6e4719b2ed05d55943d86"}
Nov 03 19:24:27.273613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.273218    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" event={"ID":"77f3a3d0-430d-4591-b943-e2231afb753e","Type":"ContainerStarted","Data":"955f4eb36756caa9d5e65cd90ae42dad0ac95d641af314decdc6cfa11d1e65aa"}
Nov 03 19:24:27.273613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.273250    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:27.275077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.275048    4944 generic.go:356] "Generic (PLEG): container finished" podID="2b07bdbe-61e3-429e-8b89-08fce8322677" containerID="31ffc99db66f78d4ecfdd40dab57693c9f66e116930a05c0292080124ecd2890" exitCode=0
Nov 03 19:24:27.275218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.275131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg" event={"ID":"2b07bdbe-61e3-429e-8b89-08fce8322677","Type":"ContainerDied","Data":"31ffc99db66f78d4ecfdd40dab57693c9f66e116930a05c0292080124ecd2890"}
Nov 03 19:24:27.276625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.276607    4944 generic.go:356] "Generic (PLEG): container finished" podID="9c8616bd-110e-4f79-b251-b4a15802f826" containerID="cd81ee739f98cedd0a9d2ecb05909810e836565af873ed258ec3ac584a554c04" exitCode=0
Nov 03 19:24:27.276734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.276685    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj" event={"ID":"9c8616bd-110e-4f79-b251-b4a15802f826","Type":"ContainerDied","Data":"cd81ee739f98cedd0a9d2ecb05909810e836565af873ed258ec3ac584a554c04"}
Nov 03 19:24:27.278255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.278234    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" event={"ID":"d5a35f35-56da-4a98-85c6-1efb41f2aadc","Type":"ContainerStarted","Data":"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"}
Nov 03 19:24:27.278337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.278273    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" event={"ID":"d5a35f35-56da-4a98-85c6-1efb41f2aadc","Type":"ContainerStarted","Data":"68abb4f1b6fb639c2b1402ba24c1d8a6401d27f928a1da47d031052cb4268b5b"}
Nov 03 19:24:27.384432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.384372    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" podStartSLOduration=1.384353643 podStartE2EDuration="1.384353643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:27.382418329 +0000 UTC m=+2365.659771937" watchObservedRunningTime="2026-11-03 19:24:27.384353643 +0000 UTC m=+2365.661707228"
Nov 03 19:24:27.460717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.460679    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:27.490600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.490571    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:27.561260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561187    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\" (UniqueName: \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\") pod \"e947238f-4bd7-4953-806f-c59544f1b53b\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") "
Nov 03 19:24:27.561432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561268    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\") pod \"b866af48-68a2-449c-afe5-1e64e05619f4\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") "
Nov 03 19:24:27.561432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561291    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\") pod \"e947238f-4bd7-4953-806f-c59544f1b53b\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") "
Nov 03 19:24:27.561432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561334    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\" (UniqueName: \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\") pod \"b866af48-68a2-449c-afe5-1e64e05619f4\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") "
Nov 03 19:24:27.561630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561433    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\") pod \"e947238f-4bd7-4953-806f-c59544f1b53b\" (UID: \"e947238f-4bd7-4953-806f-c59544f1b53b\") "
Nov 03 19:24:27.561630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561550    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\") pod \"b866af48-68a2-449c-afe5-1e64e05619f4\" (UID: \"b866af48-68a2-449c-afe5-1e64e05619f4\") "
Nov 03 19:24:27.561751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561668    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca" pod "b866af48-68a2-449c-afe5-1e64e05619f4" (UID: "b866af48-68a2-449c-afe5-1e64e05619f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:27.561751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.561685    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca" pod "e947238f-4bd7-4953-806f-c59544f1b53b" (UID: "e947238f-4bd7-4953-806f-c59544f1b53b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:27.562115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.562096    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:27.562211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.562120    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:27.563613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.563589    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms" pod "e947238f-4bd7-4953-806f-c59544f1b53b" (UID: "e947238f-4bd7-4953-806f-c59544f1b53b"). InnerVolumeSpecName "kube-api-access-9xqms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:27.563613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.563604    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg" pod "b866af48-68a2-449c-afe5-1e64e05619f4" (UID: "b866af48-68a2-449c-afe5-1e64e05619f4"). InnerVolumeSpecName "kube-api-access-ljhfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:27.564062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.564041    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca" pod "b866af48-68a2-449c-afe5-1e64e05619f4" (UID: "b866af48-68a2-449c-afe5-1e64e05619f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:27.564118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.564045    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca" pod "e947238f-4bd7-4953-806f-c59544f1b53b" (UID: "e947238f-4bd7-4953-806f-c59544f1b53b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:27.604458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.604412    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" podStartSLOduration=2.604399647 podStartE2EDuration="2.604399647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:24:27.601533061 +0000 UTC m=+2365.878886644" watchObservedRunningTime="2026-11-03 19:24:27.604399647 +0000 UTC m=+2365.881753230"
Nov 03 19:24:27.663047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.663016    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljhfg\" (UniqueName: \"kubernetes.io/projected/b866af48-68a2-449c-afe5-1e64e05619f4-kube-api-access-ljhfg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:27.663047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.663044    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e947238f-4bd7-4953-806f-c59544f1b53b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:27.663047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.663054    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b866af48-68a2-449c-afe5-1e64e05619f4-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:27.663256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:27.663063    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqms\" (UniqueName: \"kubernetes.io/projected/e947238f-4bd7-4953-806f-c59544f1b53b-kube-api-access-9xqms\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.286005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.285981    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"
Nov 03 19:24:28.286379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.286017    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr" event={"ID":"e947238f-4bd7-4953-806f-c59544f1b53b","Type":"ContainerDied","Data":"bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a"}
Nov 03 19:24:28.286379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.286074    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf3ee0ed652c70305860bac4df96befa1964c1e6a7c4104dfc21204eb8f86c3a"
Nov 03 19:24:28.287449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.287432    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"
Nov 03 19:24:28.287554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.287483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml" event={"ID":"b866af48-68a2-449c-afe5-1e64e05619f4","Type":"ContainerDied","Data":"5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869"}
Nov 03 19:24:28.287554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.287530    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bd4732ed0422024d393afba5d749591a259fe2e1137542084d1fa7848026869"
Nov 03 19:24:28.288173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.288155    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:28.440880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.440854    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:28.473051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.473011    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\") pod \"2b07bdbe-61e3-429e-8b89-08fce8322677\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") "
Nov 03 19:24:28.473251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.473230    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\") pod \"2b07bdbe-61e3-429e-8b89-08fce8322677\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") "
Nov 03 19:24:28.473335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.473294    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\" (UniqueName: \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\") pod \"2b07bdbe-61e3-429e-8b89-08fce8322677\" (UID: \"2b07bdbe-61e3-429e-8b89-08fce8322677\") "
Nov 03 19:24:28.473901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.473869    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca" pod "2b07bdbe-61e3-429e-8b89-08fce8322677" (UID: "2b07bdbe-61e3-429e-8b89-08fce8322677"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:28.476044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.476011    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca" pod "2b07bdbe-61e3-429e-8b89-08fce8322677" (UID: "2b07bdbe-61e3-429e-8b89-08fce8322677"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:28.476134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.476081    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq" pod "2b07bdbe-61e3-429e-8b89-08fce8322677" (UID: "2b07bdbe-61e3-429e-8b89-08fce8322677"). InnerVolumeSpecName "kube-api-access-mq7dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:28.477210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.477193    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:28.575024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.574949    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\" (UniqueName: \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\") pod \"9c8616bd-110e-4f79-b251-b4a15802f826\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") "
Nov 03 19:24:28.575173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575071    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\") pod \"9c8616bd-110e-4f79-b251-b4a15802f826\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") "
Nov 03 19:24:28.575173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575105    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\") pod \"9c8616bd-110e-4f79-b251-b4a15802f826\" (UID: \"9c8616bd-110e-4f79-b251-b4a15802f826\") "
Nov 03 19:24:28.575511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575480    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca" pod "9c8616bd-110e-4f79-b251-b4a15802f826" (UID: "9c8616bd-110e-4f79-b251-b4a15802f826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:24:28.575655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575612    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.575655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575636    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.575655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575652    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b07bdbe-61e3-429e-8b89-08fce8322677-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.575882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.575667    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7dq\" (UniqueName: \"kubernetes.io/projected/2b07bdbe-61e3-429e-8b89-08fce8322677-kube-api-access-mq7dq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.577289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.577265    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh" pod "9c8616bd-110e-4f79-b251-b4a15802f826" (UID: "9c8616bd-110e-4f79-b251-b4a15802f826"). InnerVolumeSpecName "kube-api-access-s8hmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:24:28.577726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.577699    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca" pod "9c8616bd-110e-4f79-b251-b4a15802f826" (UID: "9c8616bd-110e-4f79-b251-b4a15802f826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:24:28.676336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.676304    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8hmh\" (UniqueName: \"kubernetes.io/projected/9c8616bd-110e-4f79-b251-b4a15802f826-kube-api-access-s8hmh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:28.676336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:28.676335    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8616bd-110e-4f79-b251-b4a15802f826-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:24:29.295524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.295494    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"
Nov 03 19:24:29.295922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.295494    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg" event={"ID":"2b07bdbe-61e3-429e-8b89-08fce8322677","Type":"ContainerDied","Data":"bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7"}
Nov 03 19:24:29.295922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.295633    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc5d630444d8b2d553e98fb607957938efeb3978a745c1bf1342c39af99717e7"
Nov 03 19:24:29.296991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.296969    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"
Nov 03 19:24:29.297124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.297013    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj" event={"ID":"9c8616bd-110e-4f79-b251-b4a15802f826","Type":"ContainerDied","Data":"cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065"}
Nov 03 19:24:29.297124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.297062    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb439e5d9c95960b281e05d4f9c2e7ef04af68448e10389e3ddebb9dff721065"
Nov 03 19:24:29.651475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.651375    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:29.942863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:29.935233    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:30.454585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:30.454546    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:34.437574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:34.437523    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:35.224514 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:24:35.224475    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:24:38.292123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:38.292089    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:24:39.073486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:39.073455    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:39.300409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:39.300380    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:24:43.698291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:43.698216    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:48.230450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:48.230413    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:50.909480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:50.909442    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:52.182468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:52.182434    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:52.445744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:52.445654    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:53.068490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:53.068458    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:55.598033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:55.597998    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:57.728409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:57.728373    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:58.065001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:58.064962    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:58.507334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:58.507299    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:24:58.754665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:24:58.754623    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:02.267634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:02.267604    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:02.300730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:02.300696    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:25:02.306803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:02.306769    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:25:04.528409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:04.528380    4944 scope.go:122] "RemoveContainer" containerID="1416f3deacd1c01894fa7e2dbc8fcf616b8919e18578ecfb68fc4ee3ae5d90de"
Nov 03 19:25:04.538204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:04.538179    4944 scope.go:122] "RemoveContainer" containerID="52cf282ab4b4caabffd7be53a8e10138654b1055e7aba9fd8dd8e351a1310543"
Nov 03 19:25:04.546831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:04.546811    4944 scope.go:122] "RemoveContainer" containerID="4a5c0af59fc8c3f4f64a1789d0b5b3813e975961fe7e7329e7d089e9e568e2c0"
Nov 03 19:25:05.224929 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:25:05.224895    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:25:06.672880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:06.672842    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:06.918234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:06.918200    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:10.370121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:10.370092    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:10.620186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:10.620101    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:11.544348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:11.544307    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:11.804648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:11.804572    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:16.122130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:16.122097    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:20.623649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:20.623611    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:22.110025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:22.109989    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:22.439130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:22.439050    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:22.462394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:22.462354    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:25:22.462627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:22.462603    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy" containerID="cri-o://8e1b67a5a2e3701f36d5b3cf8870e8f9694011d5f260bbe53b09dc817ceacaaa" gracePeriod=30
Nov 03 19:25:22.693720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:22.693636    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:23.122622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:23.122588    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:25:23.123053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:23.122777    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" containerID="cri-o://916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf" gracePeriod=30
Nov 03 19:25:25.198356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:25.198321    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:25.494492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:25.494457    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:25.766871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:25.766766    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:26.016256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:26.016224    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:26.138881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:26.138772    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-d6xxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:25:26.138881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:26.138855    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:25:27.716459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.716427    4944 generic.go:356] "Generic (PLEG): container finished" podID="cee584a2-0005-4944-bf36-770d98f710de" containerID="8e1b67a5a2e3701f36d5b3cf8870e8f9694011d5f260bbe53b09dc817ceacaaa" exitCode=0
Nov 03 19:25:27.716809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.716498    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" event={"ID":"cee584a2-0005-4944-bf36-770d98f710de","Type":"ContainerDied","Data":"8e1b67a5a2e3701f36d5b3cf8870e8f9694011d5f260bbe53b09dc817ceacaaa"}
Nov 03 19:25:27.716809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.716558    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl" event={"ID":"cee584a2-0005-4944-bf36-770d98f710de","Type":"ContainerDied","Data":"246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08"}
Nov 03 19:25:27.716809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.716575    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246a9471a22908c051176d736e7e49188142bee6c642faee17341bc08bcb5e08"
Nov 03 19:25:27.730720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.730673    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:25:27.814815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.814759    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.814971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.814859    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.814971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.814916    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.814971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.814947    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815047    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815056    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:27.815105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815077    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815110    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815219    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:27.815266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815230    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815282    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:27.815445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815359    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\") pod \"cee584a2-0005-4944-bf36-770d98f710de\" (UID: \"cee584a2-0005-4944-bf36-770d98f710de\") "
Nov 03 19:25:27.815545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815472    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:27.815545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815473    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:27.815964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815941    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.816088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815968    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.816088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815983    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.816088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.815997    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee584a2-0005-4944-bf36-770d98f710de-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.816088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.816012    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.817213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.817192    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "kube-api-access-sg4q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:27.817323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.817283    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:25:27.817586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.817565    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:27.817692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.817593    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy" pod "cee584a2-0005-4944-bf36-770d98f710de" (UID: "cee584a2-0005-4944-bf36-770d98f710de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:27.917060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.917019    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.917060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.917048    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee584a2-0005-4944-bf36-770d98f710de-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.917060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.917059    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4q8\" (UniqueName: \"kubernetes.io/projected/cee584a2-0005-4944-bf36-770d98f710de-kube-api-access-sg4q8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:27.917060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:27.917070    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee584a2-0005-4944-bf36-770d98f710de-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.128668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.128635    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"]
Nov 03 19:25:28.129323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129303    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b866af48-68a2-449c-afe5-1e64e05619f4" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129325    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b866af48-68a2-449c-afe5-1e64e05619f4" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129349    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e947238f-4bd7-4953-806f-c59544f1b53b" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129354    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e947238f-4bd7-4953-806f-c59544f1b53b" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129361    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9c8616bd-110e-4f79-b251-b4a15802f826" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129366    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8616bd-110e-4f79-b251-b4a15802f826" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129382    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2b07bdbe-61e3-429e-8b89-08fce8322677" containerName="revoke-keys"
Nov 03 19:25:28.129387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129388    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b07bdbe-61e3-429e-8b89-08fce8322677" containerName="revoke-keys"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129417    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129422    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129503    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e947238f-4bd7-4953-806f-c59544f1b53b" containerName="revoke-keys"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129511    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8616bd-110e-4f79-b251-b4a15802f826" containerName="revoke-keys"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129519    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b07bdbe-61e3-429e-8b89-08fce8322677" containerName="revoke-keys"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129529    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee584a2-0005-4944-bf36-770d98f710de" containerName="istio-proxy"
Nov 03 19:25:28.129622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.129535    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b866af48-68a2-449c-afe5-1e64e05619f4" containerName="revoke-keys"
Nov 03 19:25:28.133347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.133327    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.182178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.182138    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:25:28.182398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.182376    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:25:28.182675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.182643    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:25:28.182917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.182901    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:25:28.186633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.186607    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"]
Nov 03 19:25:28.219710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.219674    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.219906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.219718    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sgwt\" (UniqueName: \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.219906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.219881    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.321725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.321619    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.321725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.321670    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sgwt\" (UniqueName: \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.321927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.321769    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.322749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.322715    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.324973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.324590    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.381171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.380849    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sgwt\" (UniqueName: \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\") pod \"maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.400968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.399286    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:25:28.448894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.448860    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526141    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526207    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526295    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526334    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526379    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526433    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526556    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526594    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.526665    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\") pod \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\" (UID: \"d74e3f24-3038-46ec-b27e-e72e2c1ab439\") "
Nov 03 19:25:28.527502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.527377    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:28.528183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.527956    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:28.528183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.528018    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:28.528571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.528534    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:28.528993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.528954    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:28.533256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.533232    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "kube-api-access-vjxkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:28.534107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.534081    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:28.534919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.534898    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:28.535291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.535255    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo" pod "d74e3f24-3038-46ec-b27e-e72e2c1ab439" (UID: "d74e3f24-3038-46ec-b27e-e72e2c1ab439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:25:28.628632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628598    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628629    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628643    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628651    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628661    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxkw\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-kube-api-access-vjxkw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628675    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628689    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628703    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.628916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.628723    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d74e3f24-3038-46ec-b27e-e72e2c1ab439-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:28.723801 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.723709    4944 generic.go:356] "Generic (PLEG): container finished" podID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerID="916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf" exitCode=0
Nov 03 19:25:28.723801 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.723756    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"
Nov 03 19:25:28.724246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.723804    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" event={"ID":"d74e3f24-3038-46ec-b27e-e72e2c1ab439","Type":"ContainerDied","Data":"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"}
Nov 03 19:25:28.724246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.723862    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk" event={"ID":"d74e3f24-3038-46ec-b27e-e72e2c1ab439","Type":"ContainerDied","Data":"d61fcbe1ad1fe3e9085006088702d8f83006428857b870a7a6254ca806fc4c99"}
Nov 03 19:25:28.724246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.723885    4944 scope.go:122] "RemoveContainer" containerID="916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"
Nov 03 19:25:28.724246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.724056    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"
Nov 03 19:25:28.734507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.734489    4944 scope.go:122] "RemoveContainer" containerID="916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"
Nov 03 19:25:28.734769 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:25:28.734751    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf\": container with ID starting with 916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf not found: ID does not exist" containerID="916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"
Nov 03 19:25:28.734919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.734778    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf"} err="failed to get container status \"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf\": rpc error: code = NotFound desc = could not find container \"916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf\": container with ID starting with 916874f29dc1ad323083d1204c83ef0f749f3b4d3524eff1b532cbe982301abf not found: ID does not exist"
Nov 03 19:25:28.822323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.822280    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"]
Nov 03 19:25:28.827097 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:25:28.827068    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1aa2d0a_118f_4555_b95a_0e6f2582c800.slice/crio-ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477 WatchSource:0}: Error finding container ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477: Status 404 returned error can't find the container with id ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477
Nov 03 19:25:28.889492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.889461    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:25:28.893587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.893554    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d86d1cd0-openshift-default-f565b7f45-8d6dl"]
Nov 03 19:25:28.996321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:28.996286    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:25:29.002362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.002336    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-d6xxk"]
Nov 03 19:25:29.544514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.544477    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"]
Nov 03 19:25:29.545115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.545096    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy"
Nov 03 19:25:29.545115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.545114    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy"
Nov 03 19:25:29.545292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.545237    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" containerName="istio-proxy"
Nov 03 19:25:29.548475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.548457    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.602763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.602727    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"]
Nov 03 19:25:29.638173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.638137    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.638339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.638321    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9zw6\" (UniqueName: \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.638388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.638357    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.732322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.732295    4944 generic.go:356] "Generic (PLEG): container finished" podID="d1aa2d0a-118f-4555-b95a-0e6f2582c800" containerID="3313df217ed57026447d054f9ca25ec429929dd6502b1649f34bda703d525dc5" exitCode=0
Nov 03 19:25:29.732642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.732379    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n" event={"ID":"d1aa2d0a-118f-4555-b95a-0e6f2582c800","Type":"ContainerDied","Data":"3313df217ed57026447d054f9ca25ec429929dd6502b1649f34bda703d525dc5"}
Nov 03 19:25:29.732642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.732437    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n" event={"ID":"d1aa2d0a-118f-4555-b95a-0e6f2582c800","Type":"ContainerStarted","Data":"ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477"}
Nov 03 19:25:29.739960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.739936    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9zw6\" (UniqueName: \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.740080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.739986    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.740143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.740111    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.740773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.740753    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.742215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.742195    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.797343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.797274    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9zw6\" (UniqueName: \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\") pod \"maas-api-revoke-sub-1ff2d7d2-dqb59\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.861663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.861633    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:29.939679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.939638    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee584a2-0005-4944-bf36-770d98f710de" path="/var/lib/kubelet/pods/cee584a2-0005-4944-bf36-770d98f710de/volumes"
Nov 03 19:25:29.940255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:29.940232    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74e3f24-3038-46ec-b27e-e72e2c1ab439" path="/var/lib/kubelet/pods/d74e3f24-3038-46ec-b27e-e72e2c1ab439/volumes"
Nov 03 19:25:30.234511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.232830    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"]
Nov 03 19:25:30.290101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.290076    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:30.739993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.739959    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59" event={"ID":"6676cea4-3aa2-4163-996d-27a1b2f56b8d","Type":"ContainerStarted","Data":"86b556381f783dc9b078ef0ae434b47b4c81e03e3a8a918b7821222fd57cbfa5"}
Nov 03 19:25:30.740409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.740015    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59" event={"ID":"6676cea4-3aa2-4163-996d-27a1b2f56b8d","Type":"ContainerStarted","Data":"0ba610d6d43cb50a30d50c8a4f0a505c093082a5b2974406e87f9af4b9e8d2ed"}
Nov 03 19:25:30.810750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.810715    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:30.849176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.849113    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59" podStartSLOduration=1.849098357 podStartE2EDuration="1.849098357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:30.847196655 +0000 UTC m=+2429.124550242" watchObservedRunningTime="2026-11-03 19:25:30.849098357 +0000 UTC m=+2429.126451939"
Nov 03 19:25:30.885933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.885908    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:30.956967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.956940    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\" (UniqueName: \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\") pod \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") "
Nov 03 19:25:30.957126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.957044    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\") pod \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") "
Nov 03 19:25:30.957173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.957139    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\") pod \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\" (UID: \"d1aa2d0a-118f-4555-b95a-0e6f2582c800\") "
Nov 03 19:25:30.957596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.957563    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca" pod "d1aa2d0a-118f-4555-b95a-0e6f2582c800" (UID: "d1aa2d0a-118f-4555-b95a-0e6f2582c800"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:30.959216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.959192    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca" pod "d1aa2d0a-118f-4555-b95a-0e6f2582c800" (UID: "d1aa2d0a-118f-4555-b95a-0e6f2582c800"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:30.959466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:30.959442    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt" pod "d1aa2d0a-118f-4555-b95a-0e6f2582c800" (UID: "d1aa2d0a-118f-4555-b95a-0e6f2582c800"). InnerVolumeSpecName "kube-api-access-2sgwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:31.058755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.058674    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sgwt\" (UniqueName: \"kubernetes.io/projected/d1aa2d0a-118f-4555-b95a-0e6f2582c800-kube-api-access-2sgwt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:31.058755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.058705    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:31.058755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.058716    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1aa2d0a-118f-4555-b95a-0e6f2582c800-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:31.748596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.748557    4944 generic.go:356] "Generic (PLEG): container finished" podID="6676cea4-3aa2-4163-996d-27a1b2f56b8d" containerID="86b556381f783dc9b078ef0ae434b47b4c81e03e3a8a918b7821222fd57cbfa5" exitCode=0
Nov 03 19:25:31.749041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.748637    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59" event={"ID":"6676cea4-3aa2-4163-996d-27a1b2f56b8d","Type":"ContainerDied","Data":"86b556381f783dc9b078ef0ae434b47b4c81e03e3a8a918b7821222fd57cbfa5"}
Nov 03 19:25:31.750142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.750121    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n" event={"ID":"d1aa2d0a-118f-4555-b95a-0e6f2582c800","Type":"ContainerDied","Data":"ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477"}
Nov 03 19:25:31.750250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.750144    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"
Nov 03 19:25:31.750250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:31.750158    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef872f0d8557e71d37fac40111939035af328b6fa398144e5f0f8c1581075477"
Nov 03 19:25:32.894898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.894874    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:32.980595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.980570    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\" (UniqueName: \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\") pod \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") "
Nov 03 19:25:32.980769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.980748    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\") pod \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") "
Nov 03 19:25:32.980874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.980844    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\") pod \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\" (UID: \"6676cea4-3aa2-4163-996d-27a1b2f56b8d\") "
Nov 03 19:25:32.981188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.981160    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca" pod "6676cea4-3aa2-4163-996d-27a1b2f56b8d" (UID: "6676cea4-3aa2-4163-996d-27a1b2f56b8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:32.981311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.981280    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:32.982757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.982736    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6" pod "6676cea4-3aa2-4163-996d-27a1b2f56b8d" (UID: "6676cea4-3aa2-4163-996d-27a1b2f56b8d"). InnerVolumeSpecName "kube-api-access-k9zw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:32.982873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:32.982827    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca" pod "6676cea4-3aa2-4163-996d-27a1b2f56b8d" (UID: "6676cea4-3aa2-4163-996d-27a1b2f56b8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:33.083036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:33.082954    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9zw6\" (UniqueName: \"kubernetes.io/projected/6676cea4-3aa2-4163-996d-27a1b2f56b8d-kube-api-access-k9zw6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:33.083036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:33.082982    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6676cea4-3aa2-4163-996d-27a1b2f56b8d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:33.764697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:33.764670    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"
Nov 03 19:25:33.764924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:33.764706    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59" event={"ID":"6676cea4-3aa2-4163-996d-27a1b2f56b8d","Type":"ContainerDied","Data":"0ba610d6d43cb50a30d50c8a4f0a505c093082a5b2974406e87f9af4b9e8d2ed"}
Nov 03 19:25:33.764924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:33.764780    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ba610d6d43cb50a30d50c8a4f0a505c093082a5b2974406e87f9af4b9e8d2ed"
Nov 03 19:25:35.226114 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:25:35.226073    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:25:35.336062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:35.336022    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:39.749131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:39.749075    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:40.274546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.274505    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:40.460277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.460240    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"]
Nov 03 19:25:40.461017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.460994    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6676cea4-3aa2-4163-996d-27a1b2f56b8d" containerName="revoke-keys"
Nov 03 19:25:40.461017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.461015    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6676cea4-3aa2-4163-996d-27a1b2f56b8d" containerName="revoke-keys"
Nov 03 19:25:40.461177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.461039    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d1aa2d0a-118f-4555-b95a-0e6f2582c800" containerName="revoke-keys"
Nov 03 19:25:40.461177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.461044    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1aa2d0a-118f-4555-b95a-0e6f2582c800" containerName="revoke-keys"
Nov 03 19:25:40.461177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.461123    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6676cea4-3aa2-4163-996d-27a1b2f56b8d" containerName="revoke-keys"
Nov 03 19:25:40.461177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.461135    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1aa2d0a-118f-4555-b95a-0e6f2582c800" containerName="revoke-keys"
Nov 03 19:25:40.464625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.464609    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.501550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.501523    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.501697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.501670    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb7j\" (UniqueName: \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.501748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.501732    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.530352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.530269    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:40.560623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.560594    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:25:40.560623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.560603    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:25:40.560869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.560608    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:25:40.560869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.560656    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:25:40.563929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.563909    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"]
Nov 03 19:25:40.603245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.603218    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb7j\" (UniqueName: \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.603406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.603279    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.603406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.603348    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.604018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.603997    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.605595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.605572    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.678570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.678536    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb7j\" (UniqueName: \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\") pod \"maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:40.779138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:40.779104    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:41.090676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.090563    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:41.152566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.152498    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"]
Nov 03 19:25:41.157854 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:25:41.157828    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e6c1f9_ff78_4c23_ab5a_7761a8abf877.slice/crio-9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694 WatchSource:0}: Error finding container 9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694: Status 404 returned error can't find the container with id 9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694
Nov 03 19:25:41.455874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.455773    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:41.787170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.787129    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:41.833050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.832998    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z" event={"ID":"23e6c1f9-ff78-4c23-ab5a-7761a8abf877","Type":"ContainerStarted","Data":"cc91d03ba24b9ad42d9bd8d720973633c3e9eeba46ea1bad2084d32238fc3eb2"}
Nov 03 19:25:41.833270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.833074    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z" event={"ID":"23e6c1f9-ff78-4c23-ab5a-7761a8abf877","Type":"ContainerStarted","Data":"9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694"}
Nov 03 19:25:41.945497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:41.945434    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z" podStartSLOduration=1.945415149 podStartE2EDuration="1.945415149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:41.940853148 +0000 UTC m=+2440.218206732" watchObservedRunningTime="2026-11-03 19:25:41.945415149 +0000 UTC m=+2440.222768734"
Nov 03 19:25:42.011708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:42.011669    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"]
Nov 03 19:25:42.016422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:42.016396    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-90a9bf-zc8vk"]
Nov 03 19:25:42.491999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:42.491935    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:42.841058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:42.840976    4944 generic.go:356] "Generic (PLEG): container finished" podID="23e6c1f9-ff78-4c23-ab5a-7761a8abf877" containerID="cc91d03ba24b9ad42d9bd8d720973633c3e9eeba46ea1bad2084d32238fc3eb2" exitCode=0
Nov 03 19:25:42.841058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:42.841037    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z" event={"ID":"23e6c1f9-ff78-4c23-ab5a-7761a8abf877","Type":"ContainerDied","Data":"cc91d03ba24b9ad42d9bd8d720973633c3e9eeba46ea1bad2084d32238fc3eb2"}
Nov 03 19:25:43.154084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.154002    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:43.496630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.496598    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:43.820450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.820365    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:43.938152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.938112    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f58d04-97e0-4ce5-a83f-327c5548fa2a" path="/var/lib/kubelet/pods/e4f58d04-97e0-4ce5-a83f-327c5548fa2a/volumes"
Nov 03 19:25:43.956918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.956888    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"]
Nov 03 19:25:43.994026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:43.993999    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:44.002741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.002716    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-98860e-pd48l"]
Nov 03 19:25:44.050312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.050284    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\") pod \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") "
Nov 03 19:25:44.050485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.050327    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\") pod \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") "
Nov 03 19:25:44.050537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.050500    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\" (UniqueName: \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\") pod \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\" (UID: \"23e6c1f9-ff78-4c23-ab5a-7761a8abf877\") "
Nov 03 19:25:44.050697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.050676    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca" pod "23e6c1f9-ff78-4c23-ab5a-7761a8abf877" (UID: "23e6c1f9-ff78-4c23-ab5a-7761a8abf877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:44.050990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.050970    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:44.052485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.052463    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j" pod "23e6c1f9-ff78-4c23-ab5a-7761a8abf877" (UID: "23e6c1f9-ff78-4c23-ab5a-7761a8abf877"). InnerVolumeSpecName "kube-api-access-kqb7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:44.052582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.052512    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca" pod "23e6c1f9-ff78-4c23-ab5a-7761a8abf877" (UID: "23e6c1f9-ff78-4c23-ab5a-7761a8abf877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:25:44.152261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.152175    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqb7j\" (UniqueName: \"kubernetes.io/projected/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-kube-api-access-kqb7j\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:44.152261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.152209    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23e6c1f9-ff78-4c23-ab5a-7761a8abf877-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:44.469973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.469889    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:44.858946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.858914    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"
Nov 03 19:25:44.859102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.858945    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z" event={"ID":"23e6c1f9-ff78-4c23-ab5a-7761a8abf877","Type":"ContainerDied","Data":"9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694"}
Nov 03 19:25:44.859102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:44.858997    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9294e05279338a77a20519976f0bec6ed0ee36e3f57a027085dea77cbf651694"
Nov 03 19:25:45.122851 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:45.122740    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:25:45.140681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:45.140648    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:25:45.384957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:45.384881    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:45.935772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:45.935742    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb955c15-e269-42ea-a281-72f673efd002" path="/var/lib/kubelet/pods/fb955c15-e269-42ea-a281-72f673efd002/volumes"
Nov 03 19:25:48.128646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:48.128606    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:48.604688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:48.604651    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:48.841036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:48.841000    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:25:48.841273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:48.841247    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" containerID="cri-o://3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e" gracePeriod=30
Nov 03 19:25:48.854971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:48.854911    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:49.113350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:49.113268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:49.464647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:49.464568    4944 patch_prober.go:29] interesting pod/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:25:49.464647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:49.464634    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:25:50.793241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:50.793204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:51.046744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:51.046657    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:51.298964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:51.298883    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:52.262975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:52.262940    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 19:25:52.365318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:52.365288    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 19:25:52.724167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:52.724088    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:52.972986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:52.972958    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:53.392873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.392840    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:25:53.393412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.393395    4944 cpu_manager.go:414] "containerMap: removing container" podUID="23e6c1f9-ff78-4c23-ab5a-7761a8abf877" containerName="revoke-keys"
Nov 03 19:25:53.393460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.393413    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e6c1f9-ff78-4c23-ab5a-7761a8abf877" containerName="revoke-keys"
Nov 03 19:25:53.393607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.393595    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e6c1f9-ff78-4c23-ab5a-7761a8abf877" containerName="revoke-keys"
Nov 03 19:25:53.396665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.396641    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.492502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.492464    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.492766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.492609    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.492766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.492704    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.492906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.492850    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.492957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.492913    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7qzq\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.493038 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.493014    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.493265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.493239    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.493341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.493285    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.493378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.493338    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.508622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.508592    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:25:53.594716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594681    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7qzq\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.594901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594742    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.594901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594812    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.594901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594847    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.594901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594876    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.594959    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595023    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595067    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595143    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595241    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595358    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595561    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595620    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.595778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.595753    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.597997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.597971    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.601184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.601161    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.652216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.652143    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.652662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.652634    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7qzq\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\") pod \"maas-default-gateway-openshift-default-77f556d647-lgzpd\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:53.712769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:53.712728    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:54.124872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.124844    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:25:54.186221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.186186    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:25:54.193316 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:25:54.193283    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9314ad0_e9cc_4aee_867f_2a70190ebbf0.slice/crio-e3bd6275ccfd87834f4bb88b87865d47f40299d76c58aa44b6725a6ca849f5c7 WatchSource:0}: Error finding container e3bd6275ccfd87834f4bb88b87865d47f40299d76c58aa44b6725a6ca849f5c7: Status 404 returned error can't find the container with id e3bd6275ccfd87834f4bb88b87865d47f40299d76c58aa44b6725a6ca849f5c7
Nov 03 19:25:54.196000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.195971    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:25:54.196137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.196051    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:25:54.196137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.196098    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:25:54.304700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304663    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.304915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304722    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.304915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304771    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.304915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304808    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.304915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304912    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.305280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.304981    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.305280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305035    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.305280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305108    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.305280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305150    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\") pod \"b0dd89a1-fd96-470c-b085-625fb3b12099\" (UID: \"b0dd89a1-fd96-470c-b085-625fb3b12099\") "
Nov 03 19:25:54.305815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305755    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:54.305981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305772    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:54.305981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305928    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:54.305981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305095    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:54.306170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.305995    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:25:54.307581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.307558    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:25:54.308080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.308052    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "kube-api-access-r25cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:54.308288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.308264    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:25:54.308409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.308325    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo" pod "b0dd89a1-fd96-470c-b085-625fb3b12099" (UID: "b0dd89a1-fd96-470c-b085-625fb3b12099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:25:54.407281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407247    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407278    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407288    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407297    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407305    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407315    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0dd89a1-fd96-470c-b085-625fb3b12099-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407324    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407333    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r25cm\" (UniqueName: \"kubernetes.io/projected/b0dd89a1-fd96-470c-b085-625fb3b12099-kube-api-access-r25cm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.407738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.407342    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0dd89a1-fd96-470c-b085-625fb3b12099-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:25:54.955131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.955026    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" event={"ID":"a9314ad0-e9cc-4aee-867f-2a70190ebbf0","Type":"ContainerStarted","Data":"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"}
Nov 03 19:25:54.955293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.955131    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" event={"ID":"a9314ad0-e9cc-4aee-867f-2a70190ebbf0","Type":"ContainerStarted","Data":"e3bd6275ccfd87834f4bb88b87865d47f40299d76c58aa44b6725a6ca849f5c7"}
Nov 03 19:25:54.956708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.956680    4944 generic.go:356] "Generic (PLEG): container finished" podID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerID="3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e" exitCode=0
Nov 03 19:25:54.956864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.956725    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" event={"ID":"b0dd89a1-fd96-470c-b085-625fb3b12099","Type":"ContainerDied","Data":"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"}
Nov 03 19:25:54.956864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.956736    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"
Nov 03 19:25:54.956864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.956761    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg" event={"ID":"b0dd89a1-fd96-470c-b085-625fb3b12099","Type":"ContainerDied","Data":"3d8ab134cf7c59e777ed117e97ca4707247377ebdf7d8f22aaa2b567a36d0157"}
Nov 03 19:25:54.956864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.956783    4944 scope.go:122] "RemoveContainer" containerID="3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"
Nov 03 19:25:54.967386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.967366    4944 scope.go:122] "RemoveContainer" containerID="3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"
Nov 03 19:25:54.967654 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:25:54.967634    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e\": container with ID starting with 3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e not found: ID does not exist" containerID="3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"
Nov 03 19:25:54.967722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:54.967661    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e"} err="failed to get container status \"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e\": rpc error: code = NotFound desc = could not find container \"3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e\": container with ID starting with 3d376194a7b9c5c3f5288e71e4f7059b8136f860b2de39718a890e903fba843e not found: ID does not exist"
Nov 03 19:25:55.069087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.069026    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podStartSLOduration=2.069011635 podStartE2EDuration="2.069011635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:25:55.067729498 +0000 UTC m=+2453.345083078" watchObservedRunningTime="2026-11-03 19:25:55.069011635 +0000 UTC m=+2453.346365218"
Nov 03 19:25:55.182258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.182220    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:25:55.189727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.189696    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-90a9bf-openshift-default-78cb967bb9-vvdtg"]
Nov 03 19:25:55.717826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.713139    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:25:55.733273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.733238    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:25:55.733452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.733296    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:25:55.939899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:55.939861    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" path="/var/lib/kubelet/pods/b0dd89a1-fd96-470c-b085-625fb3b12099/volumes"
Nov 03 19:25:56.732928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:56.732426    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:25:56.732928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:56.732503    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:25:57.263224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:57.263183    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing" containerID="cri-o://7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261" gracePeriod=25
Nov 03 19:25:57.366105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:57.366054    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing" containerID="cri-o://a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8" gracePeriod=25
Nov 03 19:25:57.532342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:57.532244    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:25:57.720815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:57.715117    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:25:57.720815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:57.715188    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:25:58.137347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:58.137309    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:25:58.725813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:58.721163    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:25:58.725813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:58.721221    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:25:59.143687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:59.143651    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:25:59.728813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:59.728214    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:25:59.728813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:25:59.728291    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:26:00.146270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.146231    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" containerID="cri-o://b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968" gracePeriod=45
Nov 03 19:26:00.160998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.160940    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main" containerID="cri-o://b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b" gracePeriod=45
Nov 03 19:26:00.184112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.184083    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": dial tcp 10.134.0.84:8000: connect: connection refused"
Nov 03 19:26:00.472578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.472548    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:26:00.476120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.476095    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:26:00.494028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.493991    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\" (UniqueName: \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494049    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494155    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494218    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494269    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\" (UniqueName: \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494341    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\") pod \"f635a375-39ee-4c80-914d-9819fab84b20\" (UID: \"f635a375-39ee-4c80-914d-9819fab84b20\") "
Nov 03 19:26:00.494454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494420    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.494566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494489    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.494714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494678    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.494987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494943    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.494987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494970    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.494987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.494985    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.496633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.496602    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.496730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.496665    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "kube-api-access-2wcvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:00.496778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.496752    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs" pod "f635a375-39ee-4c80-914d-9819fab84b20" (UID: "f635a375-39ee-4c80-914d-9819fab84b20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:00.596277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596239    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596308    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\" (UniqueName: \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596362    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596475    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596517    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596544    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\") pod \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\" (UID: \"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc\") "
Nov 03 19:26:00.596924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596899    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f635a375-39ee-4c80-914d-9819fab84b20-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.596924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596923    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f635a375-39ee-4c80-914d-9819fab84b20-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.597124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.596938    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wcvd\" (UniqueName: \"kubernetes.io/projected/f635a375-39ee-4c80-914d-9819fab84b20-kube-api-access-2wcvd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.597836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.597493    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.597836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.597767    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.598304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.598283    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.599448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.599422    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:00.599703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.599675    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:00.599867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.599848    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5" pod "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" (UID: "a6e3ffa1-cb61-46e7-9131-82f3bd6273dc"). InnerVolumeSpecName "kube-api-access-lhfl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697825    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697853    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhfl5\" (UniqueName: \"kubernetes.io/projected/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-kube-api-access-lhfl5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697863    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697873    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697882    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.697895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.697890    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:00.713475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.713441    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:26:00.713659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.713502    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:26:00.998782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.998750    4944 generic.go:356] "Generic (PLEG): container finished" podID="f635a375-39ee-4c80-914d-9819fab84b20" containerID="b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968" exitCode=0
Nov 03 19:26:00.998994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.998828    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"
Nov 03 19:26:00.998994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.998823    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" event={"ID":"f635a375-39ee-4c80-914d-9819fab84b20","Type":"ContainerDied","Data":"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"}
Nov 03 19:26:00.998994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.998936    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5" event={"ID":"f635a375-39ee-4c80-914d-9819fab84b20","Type":"ContainerDied","Data":"a9f38cab1b7c117bd74f73eaf2897dbc6adb4bb2e157fb6f6b00e15ec18bb38d"}
Nov 03 19:26:00.998994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:00.998958    4944 scope.go:122] "RemoveContainer" containerID="b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"
Nov 03 19:26:01.000649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.000461    4944 generic.go:356] "Generic (PLEG): container finished" podID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerID="b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b" exitCode=0
Nov 03 19:26:01.000649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.000511    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"
Nov 03 19:26:01.000649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.000548    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" event={"ID":"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc","Type":"ContainerDied","Data":"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"}
Nov 03 19:26:01.000649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.000602    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz" event={"ID":"a6e3ffa1-cb61-46e7-9131-82f3bd6273dc","Type":"ContainerDied","Data":"4427b4fb2dc3cfb23b8da7a4853700b020455b262a2aaac2a5c8daa99035232c"}
Nov 03 19:26:01.011178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.011152    4944 scope.go:122] "RemoveContainer" containerID="b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"
Nov 03 19:26:01.011675 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:01.011489    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968\": container with ID starting with b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968 not found: ID does not exist" containerID="b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"
Nov 03 19:26:01.011675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.011526    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968"} err="failed to get container status \"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968\": rpc error: code = NotFound desc = could not find container \"b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968\": container with ID starting with b58a4747ca7d20194eb168fd158872304943807cd356163224cc4b4929fd5968 not found: ID does not exist"
Nov 03 19:26:01.011675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.011549    4944 scope.go:122] "RemoveContainer" containerID="b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"
Nov 03 19:26:01.021857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.021838    4944 scope.go:122] "RemoveContainer" containerID="b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"
Nov 03 19:26:01.022154 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:01.022135    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b\": container with ID starting with b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b not found: ID does not exist" containerID="b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"
Nov 03 19:26:01.022228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.022160    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b"} err="failed to get container status \"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b\": rpc error: code = NotFound desc = could not find container \"b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b\": container with ID starting with b00161515d4fa42387e145d12d5bb649f73e3b9ac8c6b701482ccd469c81839b not found: ID does not exist"
Nov 03 19:26:01.152662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.152628    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:26:01.156887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.156858    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/distinct-w0-92596a-kserve-65484dfb85-4thw5"]
Nov 03 19:26:01.258975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.258874    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:26:01.261916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.261889    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-92596a/simulator-w0-92596a-kserve-6856c59459-szqtz"]
Nov 03 19:26:01.714032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.713941    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-lgzpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body=
Nov 03 19:26:01.714193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.714030    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused"
Nov 03 19:26:01.935911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.935879    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" path="/var/lib/kubelet/pods/a6e3ffa1-cb61-46e7-9131-82f3bd6273dc/volumes"
Nov 03 19:26:01.936351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:01.936335    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f635a375-39ee-4c80-914d-9819fab84b20" path="/var/lib/kubelet/pods/f635a375-39ee-4c80-914d-9819fab84b20/volumes"
Nov 03 19:26:02.099171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:02.099141    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:02.803766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:02.803733    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:26:02.804549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:02.804521    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:26:02.805351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:02.805287    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:26:04.590807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:04.590766    4944 scope.go:122] "RemoveContainer" containerID="108ab1e8b3fa6d3f887a7ff69f838a52c50d0dd4d6899f9a8e253517605768d1"
Nov 03 19:26:04.600325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:04.600301    4944 scope.go:122] "RemoveContainer" containerID="f909777d825ef5704b006b977bd88af471bccb51cd9b90e61b5667047e45ca8c"
Nov 03 19:26:04.608739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:04.608710    4944 scope.go:122] "RemoveContainer" containerID="fd5b7a8555a0d4ce96616812873d2836ad7b4bda41e03a2ba93f65674a48850b"
Nov 03 19:26:04.617931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:04.617907    4944 scope.go:122] "RemoveContainer" containerID="8e1b67a5a2e3701f36d5b3cf8870e8f9694011d5f260bbe53b09dc817ceacaaa"
Nov 03 19:26:05.224858 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:05.224823    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:26:06.967162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:06.967126    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:08.137653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.137612    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:26:08.497009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.496974    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:26:08.497663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497641    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main"
Nov 03 19:26:08.497779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497695    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main"
Nov 03 19:26:08.497779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497743    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy"
Nov 03 19:26:08.497779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497751    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy"
Nov 03 19:26:08.497779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497770    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main"
Nov 03 19:26:08.498027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497780    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main"
Nov 03 19:26:08.498027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497945    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0dd89a1-fd96-470c-b085-625fb3b12099" containerName="istio-proxy"
Nov 03 19:26:08.498027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497960    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e3ffa1-cb61-46e7-9131-82f3bd6273dc" containerName="main"
Nov 03 19:26:08.498027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.497980    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f635a375-39ee-4c80-914d-9819fab84b20" containerName="main"
Nov 03 19:26:08.503383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.503358    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.589880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.589846    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:26:08.605272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605242    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwcvg\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605285    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605319    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605411    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605436    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605470    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605495    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605533    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.605636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.605593    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.706887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.706850    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwcvg\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.706895    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.706938    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.706969    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.706996    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707048    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707080    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707130    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707193    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707431    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707465    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707548    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707626    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.707881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.707861    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.709262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.709245    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.709626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.709610    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.763328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.763265    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.763486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.763463    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwcvg\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\") pod \"maas-default-gateway-openshift-default-77f556d647-zdj72\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:08.816503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:08.816476    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:09.143143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:09.143062    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:26:09.245548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:09.245506    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:26:09.251317 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:26:09.251290    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3910b36b_0adb_4567_b50c_a84b2784ce45.slice/crio-46edaed97ab8e501ff67925a1626599790e273bf2b5dff207b0731bdd7aa704a WatchSource:0}: Error finding container 46edaed97ab8e501ff67925a1626599790e273bf2b5dff207b0731bdd7aa704a: Status 404 returned error can't find the container with id 46edaed97ab8e501ff67925a1626599790e273bf2b5dff207b0731bdd7aa704a
Nov 03 19:26:09.253858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:09.253829    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:26:09.253926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:09.253894    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:26:09.253926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:09.253921    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:26:10.063980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:10.063937    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" event={"ID":"3910b36b-0adb-4567-b50c-a84b2784ce45","Type":"ContainerStarted","Data":"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"}
Nov 03 19:26:10.064168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:10.063996    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" event={"ID":"3910b36b-0adb-4567-b50c-a84b2784ce45","Type":"ContainerStarted","Data":"46edaed97ab8e501ff67925a1626599790e273bf2b5dff207b0731bdd7aa704a"}
Nov 03 19:26:10.828813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:10.825981    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:10.828813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:10.826125    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:10.828813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:10.826171    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:11.829815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:11.828394    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:11.833810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:11.832923    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:11.833810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:11.832986    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:12.818230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:12.818191    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:12.818558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:12.818525    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:13.825816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:13.825269    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:13.825816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:13.825341    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:14.830814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:14.830246    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:14.830814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:14.830353    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:15.817354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:15.817320    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:15.817535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:15.817374    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:16.525262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:16.525222    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:16.817747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:16.817665    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zdj72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused" start-of-body=
Nov 03 19:26:16.817908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:16.817742    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.214:15021/healthz/ready\": dial tcp 10.134.0.214:15021: connect: connection refused"
Nov 03 19:26:17.572319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.572273    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 19:26:17.670530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.669401    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\") pod \"7a1a5694-d54b-4f71-ae61-6d57be897b37\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") "
Nov 03 19:26:17.670530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.669552    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\" (UniqueName: \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\") pod \"7a1a5694-d54b-4f71-ae61-6d57be897b37\" (UID: \"7a1a5694-d54b-4f71-ae61-6d57be897b37\") "
Nov 03 19:26:17.670909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.670834    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume" pod "7a1a5694-d54b-4f71-ae61-6d57be897b37" (UID: "7a1a5694-d54b-4f71-ae61-6d57be897b37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:17.673760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.673724    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck" pod "7a1a5694-d54b-4f71-ae61-6d57be897b37" (UID: "7a1a5694-d54b-4f71-ae61-6d57be897b37"). InnerVolumeSpecName "kube-api-access-jrxck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:17.698647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.696285    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podStartSLOduration=9.696265221 podStartE2EDuration="9.696265221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:10.243891602 +0000 UTC m=+2468.521245185" watchObservedRunningTime="2026-11-03 19:26:17.696265221 +0000 UTC m=+2475.973618805"
Nov 03 19:26:17.703806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.703768    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 19:26:17.771239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771199    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\" (UniqueName: \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\") pod \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") "
Nov 03 19:26:17.771438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771332    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\") pod \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\" (UID: \"6295d3b4-4f91-47ad-bd49-0423dd596eb8\") "
Nov 03 19:26:17.771715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771693    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume" pod "6295d3b4-4f91-47ad-bd49-0423dd596eb8" (UID: "6295d3b4-4f91-47ad-bd49-0423dd596eb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:17.771908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771890    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6295d3b4-4f91-47ad-bd49-0423dd596eb8-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:17.771984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771909    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a1a5694-d54b-4f71-ae61-6d57be897b37-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:17.771984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.771920    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrxck\" (UniqueName: \"kubernetes.io/projected/7a1a5694-d54b-4f71-ae61-6d57be897b37-kube-api-access-jrxck\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:17.773599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.773568    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4" pod "6295d3b4-4f91-47ad-bd49-0423dd596eb8" (UID: "6295d3b4-4f91-47ad-bd49-0423dd596eb8"). InnerVolumeSpecName "kube-api-access-cfnl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:17.873384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.873299    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfnl4\" (UniqueName: \"kubernetes.io/projected/6295d3b4-4f91-47ad-bd49-0423dd596eb8-kube-api-access-cfnl4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:17.976938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.976908    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:17.977467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.977448    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:17.978389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:17.978365    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:26:18.156259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.156173    4944 generic.go:356] "Generic (PLEG): container finished" podID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerID="a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8" exitCode=0
Nov 03 19:26:18.156259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.156247    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"
Nov 03 19:26:18.156467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.156269    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" event={"ID":"6295d3b4-4f91-47ad-bd49-0423dd596eb8","Type":"ContainerDied","Data":"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"}
Nov 03 19:26:18.156467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.156326    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw" event={"ID":"6295d3b4-4f91-47ad-bd49-0423dd596eb8","Type":"ContainerDied","Data":"666cfe3e2e90b0e1b7f415229ea0827fd7ff9e3b61dcb0cce6b8ababb0d2e380"}
Nov 03 19:26:18.156467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.156354    4944 scope.go:122] "RemoveContainer" containerID="a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"
Nov 03 19:26:18.157658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.157636    4944 generic.go:356] "Generic (PLEG): container finished" podID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerID="7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261" exitCode=0
Nov 03 19:26:18.157938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.157918    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"
Nov 03 19:26:18.158064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.158006    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" event={"ID":"7a1a5694-d54b-4f71-ae61-6d57be897b37","Type":"ContainerDied","Data":"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"}
Nov 03 19:26:18.158064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.158040    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg" event={"ID":"7a1a5694-d54b-4f71-ae61-6d57be897b37","Type":"ContainerDied","Data":"14bdefcf85e0e164af929372d6c8d2e7081ffc7349c32425d62ae6366016b4e5"}
Nov 03 19:26:18.168036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.168016    4944 scope.go:122] "RemoveContainer" containerID="a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"
Nov 03 19:26:18.168357 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:18.168333    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8\": container with ID starting with a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8 not found: ID does not exist" containerID="a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"
Nov 03 19:26:18.168459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.168365    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8"} err="failed to get container status \"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8\": rpc error: code = NotFound desc = could not find container \"a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8\": container with ID starting with a3fc116f971057a08e7efab6287a1e8b4b722e5b2f2ecbe1af4779e01808fbd8 not found: ID does not exist"
Nov 03 19:26:18.168459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.168388    4944 scope.go:122] "RemoveContainer" containerID="7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"
Nov 03 19:26:18.183031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.183011    4944 scope.go:122] "RemoveContainer" containerID="7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"
Nov 03 19:26:18.183318 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:18.183299    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261\": container with ID starting with 7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261 not found: ID does not exist" containerID="7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"
Nov 03 19:26:18.183389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.183326    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261"} err="failed to get container status \"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261\": rpc error: code = NotFound desc = could not find container \"7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261\": container with ID starting with 7741a667ddab454eaa63d158c510b681d0f7d97e37696830e54b8eec492ac261 not found: ID does not exist"
Nov 03 19:26:18.308005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.307975    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 19:26:18.312117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.312092    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-92596a-5497dbbcbc-tw7lw"]
Nov 03 19:26:18.415151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.415071    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 19:26:18.418984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:18.418956    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-92596a-545c494b4b-8n5hg"]
Nov 03 19:26:19.936881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:19.936843    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" path="/var/lib/kubelet/pods/6295d3b4-4f91-47ad-bd49-0423dd596eb8/volumes"
Nov 03 19:26:19.937422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:19.937292    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" path="/var/lib/kubelet/pods/7a1a5694-d54b-4f71-ae61-6d57be897b37/volumes"
Nov 03 19:26:21.179726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:21.179687    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:21.953527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:21.953493    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:22.217784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:22.217689    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:23.475097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:23.475059    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:25.906993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:25.906953    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:29.200759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:29.200704    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:29.452236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:29.452154    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.159235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:31.159197    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:31.404239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:31.404204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:33.766158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:33.766123    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:35.224971 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:35.224933    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:26:35.894924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:35.894891    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:40.491057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:40.491017    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.560503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.560467    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"]
Nov 03 19:26:44.561052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561037    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing"
Nov 03 19:26:44.561100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561053    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing"
Nov 03 19:26:44.561100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561068    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing"
Nov 03 19:26:44.561100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561073    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing"
Nov 03 19:26:44.561245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561167    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a1a5694-d54b-4f71-ae61-6d57be897b37" containerName="payload-processing"
Nov 03 19:26:44.561245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.561187    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6295d3b4-4f91-47ad-bd49-0423dd596eb8" containerName="payload-pre-processing"
Nov 03 19:26:44.564917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.564898    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.572753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.572264    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:44.613437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.613414    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"]
Nov 03 19:26:44.617350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.617318    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6r84\" (UniqueName: \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.617470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.617378    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.617571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.617545    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.618902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.618881    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.619172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.619150    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"]
Nov 03 19:26:44.656322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.656287    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:26:44.656322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.656295    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:26:44.656486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.656322    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:26:44.659839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.659820    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:26:44.671490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.671466    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"]
Nov 03 19:26:44.718306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718276    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.718453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718410    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.718453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718440    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.718576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718460    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.718685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718661    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs59g\" (UniqueName: \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.718749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718726    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6r84\" (UniqueName: \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.718967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.718948    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.720982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.720965    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.775602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.775576    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6r84\" (UniqueName: \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\") pod \"maas-api-revoke-sub-d5bb0b7f-wc8zz\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.819944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.819886    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs59g\" (UniqueName: \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.820053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.820009    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.820053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.820043    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.820864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.820835    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.822393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.822367    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.877134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.877112    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs59g\" (UniqueName: \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\") pod \"maas-api-revoke-sub-88d46164-w24zn\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:44.878944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.878928    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:44.935833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:44.935797    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:45.005272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.005249    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:45.250620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.250589    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"]
Nov 03 19:26:45.255802 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:26:45.255762    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode815a315_6d23_47c9_b353_520642ea23c5.slice/crio-3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf WatchSource:0}: Error finding container 3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf: Status 404 returned error can't find the container with id 3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf
Nov 03 19:26:45.352707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.352675    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"]
Nov 03 19:26:45.361390 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:26:45.361350    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8759967_5c8d_4762_93d5_11137e28efc3.slice/crio-8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26 WatchSource:0}: Error finding container 8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26: Status 404 returned error can't find the container with id 8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26
Nov 03 19:26:45.376932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.376907    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz" event={"ID":"e815a315-6d23-47c9-b353-520642ea23c5","Type":"ContainerStarted","Data":"3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf"}
Nov 03 19:26:45.378371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.378344    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn" event={"ID":"e8759967-5c8d-4762-93d5-11137e28efc3","Type":"ContainerStarted","Data":"8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26"}
Nov 03 19:26:45.435376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.435352    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:45.680539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:45.680503    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:46.021286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:46.021249    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:46.386211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:46.386182    4944 generic.go:356] "Generic (PLEG): container finished" podID="e815a315-6d23-47c9-b353-520642ea23c5" containerID="1ea9d35e0f1c47943db5306698e664c2cd6502b09f514a98f7340c615ee22ba0" exitCode=0
Nov 03 19:26:46.386354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:46.386262    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz" event={"ID":"e815a315-6d23-47c9-b353-520642ea23c5","Type":"ContainerDied","Data":"1ea9d35e0f1c47943db5306698e664c2cd6502b09f514a98f7340c615ee22ba0"}
Nov 03 19:26:46.387883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:46.387858    4944 generic.go:356] "Generic (PLEG): container finished" podID="e8759967-5c8d-4762-93d5-11137e28efc3" containerID="7277b42c8c197a0c389475104e331892789c967b736ce908871af7674636e074" exitCode=0
Nov 03 19:26:46.388002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:46.387946    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn" event={"ID":"e8759967-5c8d-4762-93d5-11137e28efc3","Type":"ContainerDied","Data":"7277b42c8c197a0c389475104e331892789c967b736ce908871af7674636e074"}
Nov 03 19:26:47.579705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.579673    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:47.582959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.582935    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:47.665100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665021    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\") pod \"e815a315-6d23-47c9-b353-520642ea23c5\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") "
Nov 03 19:26:47.665256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665149    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\" (UniqueName: \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\") pod \"e815a315-6d23-47c9-b353-520642ea23c5\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") "
Nov 03 19:26:47.665256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665190    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\" (UniqueName: \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\") pod \"e8759967-5c8d-4762-93d5-11137e28efc3\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") "
Nov 03 19:26:47.665372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665284    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\") pod \"e815a315-6d23-47c9-b353-520642ea23c5\" (UID: \"e815a315-6d23-47c9-b353-520642ea23c5\") "
Nov 03 19:26:47.665372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665363    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\") pod \"e8759967-5c8d-4762-93d5-11137e28efc3\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") "
Nov 03 19:26:47.665498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665380    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca" pod "e815a315-6d23-47c9-b353-520642ea23c5" (UID: "e815a315-6d23-47c9-b353-520642ea23c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:47.665498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665415    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\") pod \"e8759967-5c8d-4762-93d5-11137e28efc3\" (UID: \"e8759967-5c8d-4762-93d5-11137e28efc3\") "
Nov 03 19:26:47.665746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665723    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca" pod "e8759967-5c8d-4762-93d5-11137e28efc3" (UID: "e8759967-5c8d-4762-93d5-11137e28efc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:47.665853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665836    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:47.665934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.665854    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:47.667560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.667529    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g" pod "e8759967-5c8d-4762-93d5-11137e28efc3" (UID: "e8759967-5c8d-4762-93d5-11137e28efc3"). InnerVolumeSpecName "kube-api-access-qs59g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:47.667670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.667570    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84" pod "e815a315-6d23-47c9-b353-520642ea23c5" (UID: "e815a315-6d23-47c9-b353-520642ea23c5"). InnerVolumeSpecName "kube-api-access-j6r84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:47.667810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.667761    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca" pod "e815a315-6d23-47c9-b353-520642ea23c5" (UID: "e815a315-6d23-47c9-b353-520642ea23c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:47.667900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.667881    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca" pod "e8759967-5c8d-4762-93d5-11137e28efc3" (UID: "e8759967-5c8d-4762-93d5-11137e28efc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:47.767475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.767440    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e815a315-6d23-47c9-b353-520642ea23c5-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:47.767475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.767469    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8759967-5c8d-4762-93d5-11137e28efc3-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:47.767475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.767478    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6r84\" (UniqueName: \"kubernetes.io/projected/e815a315-6d23-47c9-b353-520642ea23c5-kube-api-access-j6r84\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:47.767709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:47.767490    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs59g\" (UniqueName: \"kubernetes.io/projected/e8759967-5c8d-4762-93d5-11137e28efc3-kube-api-access-qs59g\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:48.403716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.403683    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"
Nov 03 19:26:48.403941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.403729    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn" event={"ID":"e8759967-5c8d-4762-93d5-11137e28efc3","Type":"ContainerDied","Data":"8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26"}
Nov 03 19:26:48.403941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.403781    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d853d44c78d8f24dc7521b61243f6fa700b29ea6967dc30c6861dc2f4228a26"
Nov 03 19:26:48.405170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.405151    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"
Nov 03 19:26:48.405303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.405168    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz" event={"ID":"e815a315-6d23-47c9-b353-520642ea23c5","Type":"ContainerDied","Data":"3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf"}
Nov 03 19:26:48.405303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:48.405211    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aca566cb07125b46f402225f603f56c83633755e3fb71a29f6f1aefec1561cf"
Nov 03 19:26:49.074406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.074374    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:26:49.074904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.074582    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy" containerID="cri-o://c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41" gracePeriod=30
Nov 03 19:26:49.120203 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.120173    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:26:49.120376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.120357    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" containerName="discovery" containerID="cri-o://f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1" gracePeriod=30
Nov 03 19:26:49.375197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.375172    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:26:49.412083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.412057    4944 generic.go:356] "Generic (PLEG): container finished" podID="20f8a3ba-0552-493d-a49b-a4927964f245" containerID="f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1" exitCode=0
Nov 03 19:26:49.412219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.412095    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"
Nov 03 19:26:49.412219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.412113    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" event={"ID":"20f8a3ba-0552-493d-a49b-a4927964f245","Type":"ContainerDied","Data":"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"}
Nov 03 19:26:49.412219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.412152    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw" event={"ID":"20f8a3ba-0552-493d-a49b-a4927964f245","Type":"ContainerDied","Data":"80ea8e47066bbf4419d6cffe6c000fa6d7095ac30ad39f9495105bd46dc6a48c"}
Nov 03 19:26:49.412219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.412172    4944 scope.go:122] "RemoveContainer" containerID="f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"
Nov 03 19:26:49.422112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.422093    4944 scope.go:122] "RemoveContainer" containerID="f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"
Nov 03 19:26:49.422389 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:49.422366    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1\": container with ID starting with f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1 not found: ID does not exist" containerID="f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"
Nov 03 19:26:49.422487 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.422394    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1"} err="failed to get container status \"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1\": rpc error: code = NotFound desc = could not find container \"f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1\": container with ID starting with f4e16639d99a741ed771365a16774863c1591a16a89d8c0e4c09a954c4197dc1 not found: ID does not exist"
Nov 03 19:26:49.485541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485505    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.485763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485551    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.485763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485616    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.485763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485645    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.485763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485708    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.486087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485842    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.486087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.485940    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\") pod \"20f8a3ba-0552-493d-a49b-a4927964f245\" (UID: \"20f8a3ba-0552-493d-a49b-a4927964f245\") "
Nov 03 19:26:49.486301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.486232    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:49.486498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.486475    4944 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-ca-configmap\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.488616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.488589    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "kube-api-access-v649l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:49.489086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.488854    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:49.489086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.489050    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:49.489086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.489026    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:26:49.489387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.489364    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:49.489496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.489475    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs" pod "20f8a3ba-0552-493d-a49b-a4927964f245" (UID: "20f8a3ba-0552-493d-a49b-a4927964f245"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:49.588305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588278    4944 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/20f8a3ba-0552-493d-a49b-a4927964f245-local-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.588305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588306    4944 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-cacerts\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.588489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588318    4944 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-csr-dns-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.588489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588329    4944 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/20f8a3ba-0552-493d-a49b-a4927964f245-istio-kubeconfig\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.588489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588338    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v649l\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-kube-api-access-v649l\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.588489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.588347    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20f8a3ba-0552-493d-a49b-a4927964f245-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:49.805517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.805481    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:49.827177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.827145    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:26:49.835608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.835582    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zflrw"]
Nov 03 19:26:49.936066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:49.936031    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" path="/var/lib/kubelet/pods/20f8a3ba-0552-493d-a49b-a4927964f245/volumes"
Nov 03 19:26:50.058577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:50.058539    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:50.304397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:50.304363    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:53.082882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.074350    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:53.328925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.328893    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:53.359678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.359607    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:26:53.360164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360148    4944 cpu_manager.go:414] "containerMap: removing container" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" containerName="discovery"
Nov 03 19:26:53.360164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360163    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" containerName="discovery"
Nov 03 19:26:53.360330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360201    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e8759967-5c8d-4762-93d5-11137e28efc3" containerName="revoke-keys"
Nov 03 19:26:53.360330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360211    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8759967-5c8d-4762-93d5-11137e28efc3" containerName="revoke-keys"
Nov 03 19:26:53.360330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360227    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e815a315-6d23-47c9-b353-520642ea23c5" containerName="revoke-keys"
Nov 03 19:26:53.360330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360232    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e815a315-6d23-47c9-b353-520642ea23c5" containerName="revoke-keys"
Nov 03 19:26:53.360330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360323    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8759967-5c8d-4762-93d5-11137e28efc3" containerName="revoke-keys"
Nov 03 19:26:53.360497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360334    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e815a315-6d23-47c9-b353-520642ea23c5" containerName="revoke-keys"
Nov 03 19:26:53.360497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.360345    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="20f8a3ba-0552-493d-a49b-a4927964f245" containerName="discovery"
Nov 03 19:26:53.364833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.364815    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.413569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.413538    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-22ed3b\""
Nov 03 19:26:53.439170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.439133    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hd6g\" (UniqueName: \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.439372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.439320    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.463239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.463212    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:26:53.554119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.554073    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hd6g\" (UniqueName: \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.554305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.554186    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.555249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.555212    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.559373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.559344    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:26:53.564171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.564147    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.599504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.599472    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:26:53.613352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.612725    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-22ed3b-dockercfg-l854b\""
Nov 03 19:26:53.649738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.643178    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hd6g\" (UniqueName: \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\") pod \"payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.664546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.664499    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.664823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.664764    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kc9\" (UniqueName: \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.678850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.678818    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:53.768531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.766640    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kc9\" (UniqueName: \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.768531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.766701    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.768531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.767777    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.824114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.824079    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kc9\" (UniqueName: \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\") pod \"payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:53.882420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:53.882333    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:54.082387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.082343    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:26:54.087083 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:26:54.087054    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313f1526_a0cd_4c24_9d5b_d5918a4c379c.slice/crio-34403f94f505a7f9ecebd355d3dcd1b3f669847b903d4dc1393967eacf93b0eb WatchSource:0}: Error finding container 34403f94f505a7f9ecebd355d3dcd1b3f669847b903d4dc1393967eacf93b0eb: Status 404 returned error can't find the container with id 34403f94f505a7f9ecebd355d3dcd1b3f669847b903d4dc1393967eacf93b0eb
Nov 03 19:26:54.288474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.288432    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:26:54.294465 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:26:54.294439    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29712e8d_fac3_4466_99d0_6f38903ce8fb.slice/crio-789a101a72e3367787626b0e09a2c40c4f764df8a8b4201e284fedeb1c25dcb5 WatchSource:0}: Error finding container 789a101a72e3367787626b0e09a2c40c4f764df8a8b4201e284fedeb1c25dcb5: Status 404 returned error can't find the container with id 789a101a72e3367787626b0e09a2c40c4f764df8a8b4201e284fedeb1c25dcb5
Nov 03 19:26:54.348124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.348100    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:26:54.448177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.448096    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" event={"ID":"313f1526-a0cd-4c24-9d5b-d5918a4c379c","Type":"ContainerStarted","Data":"f6ccc253e394257039bd66ff52e31de3ca624c8007553ee7f69227089e4cafd9"}
Nov 03 19:26:54.448177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.448151    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" event={"ID":"313f1526-a0cd-4c24-9d5b-d5918a4c379c","Type":"ContainerStarted","Data":"34403f94f505a7f9ecebd355d3dcd1b3f669847b903d4dc1393967eacf93b0eb"}
Nov 03 19:26:54.448177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.448176    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:26:54.449820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.449766    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" event={"ID":"29712e8d-fac3-4466-99d0-6f38903ce8fb","Type":"ContainerStarted","Data":"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"}
Nov 03 19:26:54.449956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.449835    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" event={"ID":"29712e8d-fac3-4466-99d0-6f38903ce8fb","Type":"ContainerStarted","Data":"789a101a72e3367787626b0e09a2c40c4f764df8a8b4201e284fedeb1c25dcb5"}
Nov 03 19:26:54.449956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.449862    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:26:54.451210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.451189    4944 generic.go:356] "Generic (PLEG): container finished" podID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerID="c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41" exitCode=0
Nov 03 19:26:54.451319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.451219    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" event={"ID":"8dad976e-f8e9-438a-898a-e8e4d0df20be","Type":"ContainerDied","Data":"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"}
Nov 03 19:26:54.451319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.451252    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"
Nov 03 19:26:54.451319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.451275    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp" event={"ID":"8dad976e-f8e9-438a-898a-e8e4d0df20be","Type":"ContainerDied","Data":"a6caf01da9a15b8924382571e18c601b526948c56d2f2964b9376c5be22f7f97"}
Nov 03 19:26:54.451319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.451310    4944 scope.go:122] "RemoveContainer" containerID="c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"
Nov 03 19:26:54.461710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.461693    4944 scope.go:122] "RemoveContainer" containerID="c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"
Nov 03 19:26:54.462005 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:26:54.461985    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41\": container with ID starting with c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41 not found: ID does not exist" containerID="c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"
Nov 03 19:26:54.462082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.462016    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41"} err="failed to get container status \"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41\": rpc error: code = NotFound desc = could not find container \"c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41\": container with ID starting with c93ec96edb68411dd203448895df02de63254b6ba47af1cbd544cd560136af41 not found: ID does not exist"
Nov 03 19:26:54.475357 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.475330    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.475503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.475395    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.475740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.475711    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:26:54.477600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477579    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477650    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477702    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477731    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477780    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477835    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477886    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\") pod \"8dad976e-f8e9-438a-898a-e8e4d0df20be\" (UID: \"8dad976e-f8e9-438a-898a-e8e4d0df20be\") "
Nov 03 19:26:54.477937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477926    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:54.478176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.477942    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:54.478176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478131    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:54.478307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478172    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:54.478443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478424    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.478544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478436    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:26:54.478544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478445    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.478544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478488    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.478544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478505    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.478544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.478522    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dad976e-f8e9-438a-898a-e8e4d0df20be-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.479981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.479949    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "kube-api-access-pr95k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:54.480329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.480310    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:26:54.480604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.480585    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token" pod "8dad976e-f8e9-438a-898a-e8e4d0df20be" (UID: "8dad976e-f8e9-438a-898a-e8e4d0df20be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:26:54.579672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.579633    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.579672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.579666    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr95k\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-kube-api-access-pr95k\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.579672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.579679    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.579961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.579692    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dad976e-f8e9-438a-898a-e8e4d0df20be-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:26:54.617722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.617659    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" podStartSLOduration=1.6176392750000002 podStartE2EDuration="1.617639275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:54.616101087 +0000 UTC m=+2512.893454672" watchObservedRunningTime="2026-11-03 19:26:54.617639275 +0000 UTC m=+2512.894992859"
Nov 03 19:26:54.726362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.726256    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" podStartSLOduration=1.7262440510000001 podStartE2EDuration="1.726244051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:26:54.723373958 +0000 UTC m=+2513.000727541" watchObservedRunningTime="2026-11-03 19:26:54.726244051 +0000 UTC m=+2513.003597634"
Nov 03 19:26:54.866512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.866475    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:26:54.878330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:54.878292    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fa2878-openshift-default-6c7c8d7847-jvqgp"]
Nov 03 19:26:55.205729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:55.205697    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:26:55.934847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:55.934810    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" path="/var/lib/kubelet/pods/8dad976e-f8e9-438a-898a-e8e4d0df20be/volumes"
Nov 03 19:26:59.745637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:26:59.745606    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:01.964299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:01.964268    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:04.298903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:04.298872    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:04.674629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:04.674555    4944 scope.go:122] "RemoveContainer" containerID="4bc87290ad000f5e9dd3b5650fbdd8fdc7cb56895f0e280ea9c3d398763d8396"
Nov 03 19:27:05.224563 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:27:05.224524    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:27:05.431185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:05.431152    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:05.462285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:05.462260    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:27:05.462443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:05.462323    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:27:05.675937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:05.675907    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:08.823558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:08.823513    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:10.513021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:10.512986    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:10.760321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:10.760279    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:13.349744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:13.349710    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:17.957647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:17.957606    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:21.379715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.379679    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:27:21.380267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.380252    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy"
Nov 03 19:27:21.380321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.380267    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy"
Nov 03 19:27:21.380393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.380384    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dad976e-f8e9-438a-898a-e8e4d0df20be" containerName="istio-proxy"
Nov 03 19:27:21.385548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.385530    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.481153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.481116    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6413ba\"/\"test-model-6413ba-kserve-self-signed-certs\""
Nov 03 19:27:21.481153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.481120    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6413ba\"/\"kube-root-ca.crt\""
Nov 03 19:27:21.481367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.481177    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6413ba\"/\"openshift-service-ca.crt\""
Nov 03 19:27:21.481367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.481190    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6413ba\"/\"default-dockercfg-5bspj\""
Nov 03 19:27:21.484202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.484181    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:27:21.487739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.487715    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rsv\" (UniqueName: \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.487830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.487761    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.487830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.487803    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.487955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.487918    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.488015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.487972    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.488065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.488017    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.572650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.572613    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:21.589014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.588982    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589017    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589143    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589185    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589243    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589405    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589413    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589414    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rsv\" (UniqueName: \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.589675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.589653    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.591434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.591415    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.591716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.591697    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.646418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.646355    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rsv\" (UniqueName: \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\") pod \"test-model-6413ba-kserve-84fb4c6df-6chzd\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") " pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:21.697747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:21.697713    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:22.073694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.073659    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:27:22.078983 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:27:22.078952    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd08518d4_62bf_4e80_8a01_3c052ff3030d.slice/crio-dab77176522d93fc05a0063d7a2f58444a362e7fd7ef33bf8cc4ad6693f808e8 WatchSource:0}: Error finding container dab77176522d93fc05a0063d7a2f58444a362e7fd7ef33bf8cc4ad6693f808e8: Status 404 returned error can't find the container with id dab77176522d93fc05a0063d7a2f58444a362e7fd7ef33bf8cc4ad6693f808e8
Nov 03 19:27:22.485994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.485905    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:22.635499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.635465    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" event={"ID":"d08518d4-62bf-4e80-8a01-3c052ff3030d","Type":"ContainerStarted","Data":"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"}
Nov 03 19:27:22.635671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.635508    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" event={"ID":"d08518d4-62bf-4e80-8a01-3c052ff3030d","Type":"ContainerStarted","Data":"dab77176522d93fc05a0063d7a2f58444a362e7fd7ef33bf8cc4ad6693f808e8"}
Nov 03 19:27:22.697859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.697828    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:22.747048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:22.746999    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" podStartSLOduration=1.746985629 podStartE2EDuration="1.746985629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:22.743716904 +0000 UTC m=+2541.021070511" watchObservedRunningTime="2026-11-03 19:27:22.746985629 +0000 UTC m=+2541.024339212"
Nov 03 19:27:24.431863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:24.431828    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:25.230852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.230815    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:27:25.235712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.235686    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.288220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.288186    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:27:25.330102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.330066    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c3e530\"/\"kube-root-ca.crt\""
Nov 03 19:27:25.330251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.330107    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c3e530\"/\"openshift-service-ca.crt\""
Nov 03 19:27:25.330251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.330160    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c3e530\"/\"default-dockercfg-2dhbn\""
Nov 03 19:27:25.330666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.330649    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c3e530\"/\"test-model-c3e530-kserve-self-signed-certs\""
Nov 03 19:27:25.332438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.332412    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.332590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.332567    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56xv\" (UniqueName: \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.332667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.332640    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.332758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.332727    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.332847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.332785    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.333083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.333059    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.386685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.386648    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:25.434448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434418    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.434889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434518    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56xv\" (UniqueName: \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.434889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434557    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.434889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434578    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.434889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434606    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.434889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.434667    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.435198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.435049    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.435198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.435108    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.435198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.435128    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.436732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.436710    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.436971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.436952    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.491702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.491673    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56xv\" (UniqueName: \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\") pod \"test-model-c3e530-kserve-58c4cc6c57-zffbd\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") " pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.550719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.550691    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:25.926218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:25.926143    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:27:25.931768 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:27:25.931737    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680f8fd7_dd37_47d7_868f_2878bfd4ff90.slice/crio-5d41ca86c7f8c761d1b832a10481975846f80434fc7c16ad07e3053970d47920 WatchSource:0}: Error finding container 5d41ca86c7f8c761d1b832a10481975846f80434fc7c16ad07e3053970d47920: Status 404 returned error can't find the container with id 5d41ca86c7f8c761d1b832a10481975846f80434fc7c16ad07e3053970d47920
Nov 03 19:27:26.459398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:26.459309    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:26.675447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:26.675407    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" event={"ID":"680f8fd7-dd37-47d7-868f-2878bfd4ff90","Type":"ContainerStarted","Data":"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"}
Nov 03 19:27:26.675644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:26.675460    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" event={"ID":"680f8fd7-dd37-47d7-868f-2878bfd4ff90","Type":"ContainerStarted","Data":"5d41ca86c7f8c761d1b832a10481975846f80434fc7c16ad07e3053970d47920"}
Nov 03 19:27:26.786751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:26.786695    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" podStartSLOduration=1.786679576 podStartE2EDuration="1.786679576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:26.784154085 +0000 UTC m=+2545.061507669" watchObservedRunningTime="2026-11-03 19:27:26.786679576 +0000 UTC m=+2545.064033174"
Nov 03 19:27:27.551027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:27.550995    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:27.907615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:27.907539    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:28.202031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:28.201942    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:28.272244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:28.272207    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"]
Nov 03 19:27:28.317969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:28.317935    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d2cbe3-2549n"]
Nov 03 19:27:28.813502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:28.813450    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:29.934778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:29.934744    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1aa2d0a-118f-4555-b95a-0e6f2582c800" path="/var/lib/kubelet/pods/d1aa2d0a-118f-4555-b95a-0e6f2582c800/volumes"
Nov 03 19:27:31.504366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:31.504008    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:31.698473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:31.698444    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:31.711831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:31.711780    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:31.723112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:31.723088    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:27:33.121054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:33.121021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:34.763514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:34.763479    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:35.009370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.009334    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:35.225336 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:27:35.225255    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:27:35.256602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.256555    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:35.551442 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.551411    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:35.564091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.564066    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:35.587795 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.587769    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"]
Nov 03 19:27:35.593044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.593025    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.645698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.645471    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"]
Nov 03 19:27:35.656956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.656921    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.657117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.657007    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dlx\" (UniqueName: \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.657188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.657118    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.658894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.658867    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:27:35.659044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.658932    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:27:35.659112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.659047    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:27:35.659167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.659114    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:27:35.748857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.748830    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:27:35.758396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.758368    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.758513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.758482    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.758576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.758543    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dlx\" (UniqueName: \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.759143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.759117    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.760659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.760638    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.817486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.817416    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dlx\" (UniqueName: \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\") pod \"maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:35.907249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:35.907219    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:36.295119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:36.295083    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"]
Nov 03 19:27:36.300144 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:27:36.300107    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103fd114_1b52_4d3a_ab08_aa8c72adf652.slice/crio-8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98 WatchSource:0}: Error finding container 8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98: Status 404 returned error can't find the container with id 8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98
Nov 03 19:27:36.745777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:36.745742    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5" event={"ID":"103fd114-1b52-4d3a-ab08-aa8c72adf652","Type":"ContainerStarted","Data":"fd5ba0c5a7177fa098335d42992bf45d6591f07e8c598b4060a6fa480a6e2036"}
Nov 03 19:27:36.745974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:36.745825    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5" event={"ID":"103fd114-1b52-4d3a-ab08-aa8c72adf652","Type":"ContainerStarted","Data":"8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98"}
Nov 03 19:27:36.870345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:36.869473    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5" podStartSLOduration=1.869456788 podStartE2EDuration="1.869456788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:27:36.869004908 +0000 UTC m=+2555.146358493" watchObservedRunningTime="2026-11-03 19:27:36.869456788 +0000 UTC m=+2555.146810371"
Nov 03 19:27:37.753488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:37.753451    4944 generic.go:356] "Generic (PLEG): container finished" podID="103fd114-1b52-4d3a-ab08-aa8c72adf652" containerID="fd5ba0c5a7177fa098335d42992bf45d6591f07e8c598b4060a6fa480a6e2036" exitCode=0
Nov 03 19:27:37.753665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:37.753537    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5" event={"ID":"103fd114-1b52-4d3a-ab08-aa8c72adf652","Type":"ContainerDied","Data":"fd5ba0c5a7177fa098335d42992bf45d6591f07e8c598b4060a6fa480a6e2036"}
Nov 03 19:27:38.179561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.179468    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:38.913190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.913166    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:38.997221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.997194    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\") pod \"103fd114-1b52-4d3a-ab08-aa8c72adf652\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") "
Nov 03 19:27:38.997370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.997244    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\") pod \"103fd114-1b52-4d3a-ab08-aa8c72adf652\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") "
Nov 03 19:27:38.997370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.997337    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\" (UniqueName: \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\") pod \"103fd114-1b52-4d3a-ab08-aa8c72adf652\" (UID: \"103fd114-1b52-4d3a-ab08-aa8c72adf652\") "
Nov 03 19:27:38.997544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.997521    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca" pod "103fd114-1b52-4d3a-ab08-aa8c72adf652" (UID: "103fd114-1b52-4d3a-ab08-aa8c72adf652"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:27:38.998112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.997929    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:38.999433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.999409    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca" pod "103fd114-1b52-4d3a-ab08-aa8c72adf652" (UID: "103fd114-1b52-4d3a-ab08-aa8c72adf652"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:27:38.999533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:38.999441    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx" pod "103fd114-1b52-4d3a-ab08-aa8c72adf652" (UID: "103fd114-1b52-4d3a-ab08-aa8c72adf652"). InnerVolumeSpecName "kube-api-access-w4dlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:27:39.098903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.098826    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/103fd114-1b52-4d3a-ab08-aa8c72adf652-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:39.098903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.098854    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4dlx\" (UniqueName: \"kubernetes.io/projected/103fd114-1b52-4d3a-ab08-aa8c72adf652-kube-api-access-w4dlx\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:39.769611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.769576    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"
Nov 03 19:27:39.769611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.769603    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5" event={"ID":"103fd114-1b52-4d3a-ab08-aa8c72adf652","Type":"ContainerDied","Data":"8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98"}
Nov 03 19:27:39.770091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.769654    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df5e48d22cf491ea7337efa0fe86f0f40ca2447576095411183e24e4bcd6a98"
Nov 03 19:27:39.805991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:39.805967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:42.754637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:42.754539    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:47.200674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:47.200638    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:27:47.291681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:47.291640    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:27:47.346522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:47.346480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:47.661642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:47.661606    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:47.905457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:47.905423    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:51.786913 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:51.786881    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:52.040302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.040205    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:52.201853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.201814    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" podUID="826aa391-154d-40a3-9d0e-e5840556f19e" containerName="payload-processing" containerID="cri-o://baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1" gracePeriod=25
Nov 03 19:27:52.287123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.287090    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:52.292998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.292918    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" podUID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" containerName="payload-pre-processing" containerID="cri-o://b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe" gracePeriod=25
Nov 03 19:27:52.567481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.567423    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:27:52.588368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.588347    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:27:52.672111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.672080    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\") pod \"826aa391-154d-40a3-9d0e-e5840556f19e\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") "
Nov 03 19:27:52.672308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.672296    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\" (UniqueName: \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\") pod \"826aa391-154d-40a3-9d0e-e5840556f19e\" (UID: \"826aa391-154d-40a3-9d0e-e5840556f19e\") "
Nov 03 19:27:52.672540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.672511    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume" pod "826aa391-154d-40a3-9d0e-e5840556f19e" (UID: "826aa391-154d-40a3-9d0e-e5840556f19e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:27:52.674303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.674279    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl" pod "826aa391-154d-40a3-9d0e-e5840556f19e" (UID: "826aa391-154d-40a3-9d0e-e5840556f19e"). InnerVolumeSpecName "kube-api-access-nxbtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:27:52.773654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.773623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\") pod \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") "
Nov 03 19:27:52.773887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.773687    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\" (UniqueName: \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\") pod \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\" (UID: \"bf27e9f8-9cef-482b-b3f7-a5f6570252cb\") "
Nov 03 19:27:52.774058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.774035    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume" pod "bf27e9f8-9cef-482b-b3f7-a5f6570252cb" (UID: "bf27e9f8-9cef-482b-b3f7-a5f6570252cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:27:52.774227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.774211    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:52.774278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.774230    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxbtl\" (UniqueName: \"kubernetes.io/projected/826aa391-154d-40a3-9d0e-e5840556f19e-kube-api-access-nxbtl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:52.774278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.774244    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/826aa391-154d-40a3-9d0e-e5840556f19e-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:52.775811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.775775    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv" pod "bf27e9f8-9cef-482b-b3f7-a5f6570252cb" (UID: "bf27e9f8-9cef-482b-b3f7-a5f6570252cb"). InnerVolumeSpecName "kube-api-access-prcrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:27:52.863511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.863432    4944 generic.go:356] "Generic (PLEG): container finished" podID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" containerID="b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe" exitCode=0
Nov 03 19:27:52.863511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.863476    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"
Nov 03 19:27:52.864050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.863517    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" event={"ID":"bf27e9f8-9cef-482b-b3f7-a5f6570252cb","Type":"ContainerDied","Data":"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"}
Nov 03 19:27:52.864050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.863568    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx" event={"ID":"bf27e9f8-9cef-482b-b3f7-a5f6570252cb","Type":"ContainerDied","Data":"3732f8a1787167b7192b945e350aa4263c4c89fb46e9420fde2742fc33e9b64c"}
Nov 03 19:27:52.864050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.863591    4944 scope.go:122] "RemoveContainer" containerID="b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"
Nov 03 19:27:52.865148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.865123    4944 generic.go:356] "Generic (PLEG): container finished" podID="826aa391-154d-40a3-9d0e-e5840556f19e" containerID="baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1" exitCode=0
Nov 03 19:27:52.865255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.865171    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"
Nov 03 19:27:52.865255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.865206    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" event={"ID":"826aa391-154d-40a3-9d0e-e5840556f19e","Type":"ContainerDied","Data":"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"}
Nov 03 19:27:52.865255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.865252    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l" event={"ID":"826aa391-154d-40a3-9d0e-e5840556f19e","Type":"ContainerDied","Data":"cd53d7e8f936c944d2b6b84b48097113c29d657519c55b3590a6108e01580674"}
Nov 03 19:27:52.874957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.874937    4944 scope.go:122] "RemoveContainer" containerID="b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"
Nov 03 19:27:52.875363 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:27:52.875250    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe\": container with ID starting with b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe not found: ID does not exist" containerID="b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"
Nov 03 19:27:52.875363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.875285    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe"} err="failed to get container status \"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe\": rpc error: code = NotFound desc = could not find container \"b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe\": container with ID starting with b851b52695ac7ce3fc78de351906f9aeefef5a6c6246a23d4753eb2941165fbe not found: ID does not exist"
Nov 03 19:27:52.875363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.875310    4944 scope.go:122] "RemoveContainer" containerID="baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"
Nov 03 19:27:52.875973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.875643    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prcrv\" (UniqueName: \"kubernetes.io/projected/bf27e9f8-9cef-482b-b3f7-a5f6570252cb-kube-api-access-prcrv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:27:52.883985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.883968    4944 scope.go:122] "RemoveContainer" containerID="baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"
Nov 03 19:27:52.884240 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:27:52.884222    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1\": container with ID starting with baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1 not found: ID does not exist" containerID="baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"
Nov 03 19:27:52.884289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:52.884246    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1"} err="failed to get container status \"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1\": rpc error: code = NotFound desc = could not find container \"baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1\": container with ID starting with baa0604f56fb5e9d5617d56e25e335ffff6b454c7623a4127d79255a74c786d1 not found: ID does not exist"
Nov 03 19:27:53.017970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.017933    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:27:53.025980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.025938    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-efe6c7-69f77bd84-q4v2l"]
Nov 03 19:27:53.124867 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.124760    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:27:53.128723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.128689    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-efe6c7-85fcf859dc-2psdx"]
Nov 03 19:27:53.933832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.933772    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="826aa391-154d-40a3-9d0e-e5840556f19e" path="/var/lib/kubelet/pods/826aa391-154d-40a3-9d0e-e5840556f19e/volumes"
Nov 03 19:27:53.934230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:53.934215    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" path="/var/lib/kubelet/pods/bf27e9f8-9cef-482b-b3f7-a5f6570252cb/volumes"
Nov 03 19:27:55.940058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:55.940018    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:57.491332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:57.491293    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:57.765327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:57.765235    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:57.867519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:57.867484    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"]
Nov 03 19:27:57.913650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:57.913617    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-92596a-mg87z"]
Nov 03 19:27:57.934025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:57.933989    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e6c1f9-ff78-4c23-ab5a-7761a8abf877" path="/var/lib/kubelet/pods/23e6c1f9-ff78-4c23-ab5a-7761a8abf877/volumes"
Nov 03 19:27:58.010863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:58.010830    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:27:58.388804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:27:58.388767    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:02.769736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:02.769702    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:04.336361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.336325    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:04.559086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.559048    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:28:04.559324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.559298    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" containerID="cri-o://62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82" gracePeriod=30
Nov 03 19:28:04.586661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.586598    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:04.715971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.715940    4944 scope.go:122] "RemoveContainer" containerID="bfe760fa8feced3aa40326748ac90bca4b28d7a3f94fc5e34ad95f75b3763edd"
Nov 03 19:28:04.726045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.726025    4944 scope.go:122] "RemoveContainer" containerID="a373676049d061222268fc937c23447f02bc7ba90630acf07b8d6c4f9da234cf"
Nov 03 19:28:04.831420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:04.831384    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:05.224515 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:05.224481    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:28:05.342826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:05.342773    4944 patch_prober.go:29] interesting pod/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:28:05.343212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:05.342857    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:28:07.393251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:07.393206    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:09.825949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.825888    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:28:09.885414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885384    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885426    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885480    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885528    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885559    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885690    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885742    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.885840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885819    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\") pod \"e771afd2-b988-461e-9626-2eb1bb142cd7\" (UID: \"e771afd2-b988-461e-9626-2eb1bb142cd7\") "
Nov 03 19:28:09.886013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885935    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:09.886013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.885950    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:09.886180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886161    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:09.886274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886252    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:28:09.886381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886362    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:09.886503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886486    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.886571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886509    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.886571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886523    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e771afd2-b988-461e-9626-2eb1bb142cd7-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.886571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886537    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.886571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.886551    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.887590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.887560    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:09.887824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.887805    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:28:09.888066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.888050    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:09.888173 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.888153    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7" pod "e771afd2-b988-461e-9626-2eb1bb142cd7" (UID: "e771afd2-b988-461e-9626-2eb1bb142cd7"). InnerVolumeSpecName "kube-api-access-bt8m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:09.985755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.985723    4944 generic.go:356] "Generic (PLEG): container finished" podID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerID="62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82" exitCode=0
Nov 03 19:28:09.985970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.985899    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" event={"ID":"e771afd2-b988-461e-9626-2eb1bb142cd7","Type":"ContainerDied","Data":"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"}
Nov 03 19:28:09.985970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.985938    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq" event={"ID":"e771afd2-b988-461e-9626-2eb1bb142cd7","Type":"ContainerDied","Data":"19547e1689ad81ac3f6c4cad13fc9cec7ea16f81be11faaa842f4a22a9358c9f"}
Nov 03 19:28:09.985970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.985964    4944 scope.go:122] "RemoveContainer" containerID="62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"
Nov 03 19:28:09.986223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.986112    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"
Nov 03 19:28:09.987643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.987618    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.987782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.987648    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt8m7\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-kube-api-access-bt8m7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.987782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.987667    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.987782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.987683    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e771afd2-b988-461e-9626-2eb1bb142cd7-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:09.997122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.997097    4944 scope.go:122] "RemoveContainer" containerID="62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"
Nov 03 19:28:09.997404 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:09.997386    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82\": container with ID starting with 62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82 not found: ID does not exist" containerID="62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"
Nov 03 19:28:09.997506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:09.997409    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82"} err="failed to get container status \"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82\": rpc error: code = NotFound desc = could not find container \"62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82\": container with ID starting with 62ba3f0cd6de0f6e02682bd6440341ca7af7415bba0a13922716ed7c09350a82 not found: ID does not exist"
Nov 03 19:28:10.142488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:10.142405    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:28:10.148293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:10.148257    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-92596a-openshift-default-69db5799fc-5q2vq"]
Nov 03 19:28:11.889954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:11.889919    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:11.933638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:11.933609    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" path="/var/lib/kubelet/pods/e771afd2-b988-461e-9626-2eb1bb142cd7/volumes"
Nov 03 19:28:16.755217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:16.755181    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:21.525116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:21.525058    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:26.147059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.147017    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:26.714394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714360    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"]
Nov 03 19:28:26.714907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714892    4944 cpu_manager.go:414] "containerMap: removing container" podUID="103fd114-1b52-4d3a-ab08-aa8c72adf652" containerName="revoke-keys"
Nov 03 19:28:26.714972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714907    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103fd114-1b52-4d3a-ab08-aa8c72adf652" containerName="revoke-keys"
Nov 03 19:28:26.714972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714939    4944 cpu_manager.go:414] "containerMap: removing container" podUID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" containerName="payload-pre-processing"
Nov 03 19:28:26.714972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714946    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" containerName="payload-pre-processing"
Nov 03 19:28:26.714972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714958    4944 cpu_manager.go:414] "containerMap: removing container" podUID="826aa391-154d-40a3-9d0e-e5840556f19e" containerName="payload-processing"
Nov 03 19:28:26.714972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714964    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="826aa391-154d-40a3-9d0e-e5840556f19e" containerName="payload-processing"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714976    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.714981    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.715077    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf27e9f8-9cef-482b-b3f7-a5f6570252cb" containerName="payload-pre-processing"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.715087    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="826aa391-154d-40a3-9d0e-e5840556f19e" containerName="payload-processing"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.715096    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e771afd2-b988-461e-9626-2eb1bb142cd7" containerName="istio-proxy"
Nov 03 19:28:26.715235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.715104    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="103fd114-1b52-4d3a-ab08-aa8c72adf652" containerName="revoke-keys"
Nov 03 19:28:26.718421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.718406    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.770450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.770410    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.770624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.770459    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.770624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.770599    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.814682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.814648    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:28:26.814875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.814710    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:28:26.814875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.814728    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:28:26.814875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.814851    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:28:26.818059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.818026    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"]
Nov 03 19:28:26.871569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.871540    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.871743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.871651    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.871743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.871676    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.872273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.872248    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.873967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.873949    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:26.929350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:26.929323    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\") pod \"maas-api-revoke-sub-efc45999-vmvxz\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:27.032914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:27.032884    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:27.414078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:27.413992    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"]
Nov 03 19:28:27.419853 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:28:27.419820    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dffdf7_4e77_4d89_8d16_c0d141d96890.slice/crio-6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a WatchSource:0}: Error finding container 6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a: Status 404 returned error can't find the container with id 6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a
Nov 03 19:28:28.131953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.131913    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz" event={"ID":"64dffdf7-4e77-4d89-8d16-c0d141d96890","Type":"ContainerStarted","Data":"6451742ff7e151d62a3a4f169f4a97b0e6d5c388bc5380de7a3f28f29f5b0453"}
Nov 03 19:28:28.132122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.131985    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz" event={"ID":"64dffdf7-4e77-4d89-8d16-c0d141d96890","Type":"ContainerStarted","Data":"6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a"}
Nov 03 19:28:28.230265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.230230    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:28:28.234577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.234551    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.273949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.273899    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz" podStartSLOduration=2.273884275 podStartE2EDuration="2.273884275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:28:28.272210252 +0000 UTC m=+2606.549563835" watchObservedRunningTime="2026-11-03 19:28:28.273884275 +0000 UTC m=+2606.551237858"
Nov 03 19:28:28.283272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.283248    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c3e530-openshift-default-dockercfg-jtvxs\""
Nov 03 19:28:28.292298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292256    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292332    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292372    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292412    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292511    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292553    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292600    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292658    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfg4v\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292691    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.292833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.292728    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:28:28.394246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394167    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394210    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394323    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394399    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394442    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394486    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394521    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394558    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfg4v\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394588    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394806    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.394966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394895    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.395009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.394965    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.395055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.395041    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.395200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.395176    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.396513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.396495    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.396736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.396719    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.451245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.451201    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.451718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.451605    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfg4v\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\") pod \"e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") " pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.549067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.549034    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:28.975348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.975293    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:28:28.980390 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:28:28.980361    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b728339_4a94_4aa4_b75f_c1b16589d9ce.slice/crio-0aff9db4e52cf717f0302325ea18df0416e24730332fb1d87c0a577bbdfb4bbe WatchSource:0}: Error finding container 0aff9db4e52cf717f0302325ea18df0416e24730332fb1d87c0a577bbdfb4bbe: Status 404 returned error can't find the container with id 0aff9db4e52cf717f0302325ea18df0416e24730332fb1d87c0a577bbdfb4bbe
Nov 03 19:28:28.983068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.983038    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:28:28.983135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.983118    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:28:28.983180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:28.983148    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:28:29.141480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.141433    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" event={"ID":"2b728339-4a94-4aa4-b75f-c1b16589d9ce","Type":"ContainerStarted","Data":"ba9f7a800b031b9926e535c13c132f7e03bf2a527bec062d1bb9e25394e077fa"}
Nov 03 19:28:29.141716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.141503    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" event={"ID":"2b728339-4a94-4aa4-b75f-c1b16589d9ce","Type":"ContainerStarted","Data":"0aff9db4e52cf717f0302325ea18df0416e24730332fb1d87c0a577bbdfb4bbe"}
Nov 03 19:28:29.143379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.143348    4944 generic.go:356] "Generic (PLEG): container finished" podID="64dffdf7-4e77-4d89-8d16-c0d141d96890" containerID="6451742ff7e151d62a3a4f169f4a97b0e6d5c388bc5380de7a3f28f29f5b0453" exitCode=0
Nov 03 19:28:29.143481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.143413    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz" event={"ID":"64dffdf7-4e77-4d89-8d16-c0d141d96890","Type":"ContainerDied","Data":"6451742ff7e151d62a3a4f169f4a97b0e6d5c388bc5380de7a3f28f29f5b0453"}
Nov 03 19:28:29.257122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.257063    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podStartSLOduration=1.2570480339999999 podStartE2EDuration="1.257048034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:28:29.253407642 +0000 UTC m=+2607.530761226" watchObservedRunningTime="2026-11-03 19:28:29.257048034 +0000 UTC m=+2607.534401614"
Nov 03 19:28:29.549252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:29.549170    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:30.361221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.361190    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:30.441352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.441257    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\") pod \"64dffdf7-4e77-4d89-8d16-c0d141d96890\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") "
Nov 03 19:28:30.441533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.441416    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\") pod \"64dffdf7-4e77-4d89-8d16-c0d141d96890\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") "
Nov 03 19:28:30.441533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.441468    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\") pod \"64dffdf7-4e77-4d89-8d16-c0d141d96890\" (UID: \"64dffdf7-4e77-4d89-8d16-c0d141d96890\") "
Nov 03 19:28:30.441649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.441611    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca" pod "64dffdf7-4e77-4d89-8d16-c0d141d96890" (UID: "64dffdf7-4e77-4d89-8d16-c0d141d96890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:28:30.441901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.441874    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:30.443565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.443539    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6" pod "64dffdf7-4e77-4d89-8d16-c0d141d96890" (UID: "64dffdf7-4e77-4d89-8d16-c0d141d96890"). InnerVolumeSpecName "kube-api-access-qpkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:30.443668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.443630    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca" pod "64dffdf7-4e77-4d89-8d16-c0d141d96890" (UID: "64dffdf7-4e77-4d89-8d16-c0d141d96890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:28:30.543734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.543695    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64dffdf7-4e77-4d89-8d16-c0d141d96890-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:30.543734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.543732    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpkc6\" (UniqueName: \"kubernetes.io/projected/64dffdf7-4e77-4d89-8d16-c0d141d96890-kube-api-access-qpkc6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:30.550818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.550762    4944 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 19:28:30.551225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.550849    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 19:28:30.797010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:30.796963    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:31.159085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.158992    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"
Nov 03 19:28:31.159085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.159011    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz" event={"ID":"64dffdf7-4e77-4d89-8d16-c0d141d96890","Type":"ContainerDied","Data":"6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a"}
Nov 03 19:28:31.159085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.159067    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6edf421ad302d30881e5a5623b2305a42e657fc4f699a4c50d7ab71b3ee7251a"
Nov 03 19:28:31.246089 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:31.246056    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dffdf7_4e77_4d89_8d16_c0d141d96890.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:28:31.550005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.549965    4944 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 19:28:31.550181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.550021    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 19:28:31.866815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:31.866706    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:32.550182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:32.550146    4944 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 19:28:32.550372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:32.550211    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 19:28:33.550372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:33.550341    4944 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 19:28:33.550800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:33.550390    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 19:28:34.549733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:34.549693    4944 patch_prober.go:29] interesting pod/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 03 19:28:34.549936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:34.549751    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 03 19:28:35.224774 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:35.224731    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:28:35.316216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:35.316172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:35.553888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:35.553856    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:35.554354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:35.554329    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:35.555055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:35.555035    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:28:37.823460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:37.823422    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 19:28:37.823943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:37.823634    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy" containerID="cri-o://28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a" gracePeriod=30
Nov 03 19:28:38.719274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:38.719239    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:39.595025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.594984    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"]
Nov 03 19:28:39.595544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.595529    4944 cpu_manager.go:414] "containerMap: removing container" podUID="64dffdf7-4e77-4d89-8d16-c0d141d96890" containerName="revoke-keys"
Nov 03 19:28:39.595594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.595544    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dffdf7-4e77-4d89-8d16-c0d141d96890" containerName="revoke-keys"
Nov 03 19:28:39.595675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.595664    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dffdf7-4e77-4d89-8d16-c0d141d96890" containerName="revoke-keys"
Nov 03 19:28:39.599189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.599170    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.647998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.647958    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.648181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.648026    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.648181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.648123    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzt5k\" (UniqueName: \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.648932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.648908    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:28:39.695128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.695082    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:28:39.695128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.695093    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:28:39.695128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.695119    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:28:39.698473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.698449    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"]
Nov 03 19:28:39.749243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.749210    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.749388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.749260    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzt5k\" (UniqueName: \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.749430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.749404    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.749960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.749937    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.752054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.752029    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.806383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.806351    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzt5k\" (UniqueName: \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\") pod \"maas-api-revoke-sub-db361c36-qlpcg\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:39.913278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:39.913195    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:40.015485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:40.015390    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:40.291381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:40.291350    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"]
Nov 03 19:28:40.296234 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:28:40.296208    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34c2d52_6f21_4157_944d_9a95f76433a5.slice/crio-c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb WatchSource:0}: Error finding container c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb: Status 404 returned error can't find the container with id c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb
Nov 03 19:28:41.227432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:41.227406    4944 generic.go:356] "Generic (PLEG): container finished" podID="f34c2d52-6f21-4157-944d-9a95f76433a5" containerID="e0df815b604af5fe5e339e6b61f870acc47302ffc37d04b2841671ede9675098" exitCode=0
Nov 03 19:28:41.227755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:41.227446    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg" event={"ID":"f34c2d52-6f21-4157-944d-9a95f76433a5","Type":"ContainerDied","Data":"e0df815b604af5fe5e339e6b61f870acc47302ffc37d04b2841671ede9675098"}
Nov 03 19:28:41.227755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:41.227483    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg" event={"ID":"f34c2d52-6f21-4157-944d-9a95f76433a5","Type":"ContainerStarted","Data":"c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb"}
Nov 03 19:28:42.381433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.381400    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:42.481561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.481524    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\") pod \"f34c2d52-6f21-4157-944d-9a95f76433a5\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") "
Nov 03 19:28:42.481741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.481596    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\" (UniqueName: \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\") pod \"f34c2d52-6f21-4157-944d-9a95f76433a5\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") "
Nov 03 19:28:42.481741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.481726    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\") pod \"f34c2d52-6f21-4157-944d-9a95f76433a5\" (UID: \"f34c2d52-6f21-4157-944d-9a95f76433a5\") "
Nov 03 19:28:42.482109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.482086    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca" pod "f34c2d52-6f21-4157-944d-9a95f76433a5" (UID: "f34c2d52-6f21-4157-944d-9a95f76433a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:28:42.482257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.482240    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:42.483554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.483537    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca" pod "f34c2d52-6f21-4157-944d-9a95f76433a5" (UID: "f34c2d52-6f21-4157-944d-9a95f76433a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:28:42.483762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.483748    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k" pod "f34c2d52-6f21-4157-944d-9a95f76433a5" (UID: "f34c2d52-6f21-4157-944d-9a95f76433a5"). InnerVolumeSpecName "kube-api-access-xzt5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:42.583739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.583701    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f34c2d52-6f21-4157-944d-9a95f76433a5-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:42.583739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:42.583735    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzt5k\" (UniqueName: \"kubernetes.io/projected/f34c2d52-6f21-4157-944d-9a95f76433a5-kube-api-access-xzt5k\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.076505 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.076479    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 19:28:43.190874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.190771    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.190874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.190860    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.190958    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191016    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191053    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191178    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191210    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:43.191287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191259    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191297    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191297    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:43.191443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191335    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\") pod \"b540b40d-ecb8-4938-803c-38a23bf724fd\" (UID: \"b540b40d-ecb8-4938-803c-38a23bf724fd\") "
Nov 03 19:28:43.191585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191444    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:43.191640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191575    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:28:43.191640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191617    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:43.191909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191884    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.191909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191905    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.192129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191915    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.192129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191925    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b540b40d-ecb8-4938-803c-38a23bf724fd-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.192129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.191934    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.193245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.193224    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:43.193461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.193445    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "kube-api-access-wr5n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:28:43.193524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.193457    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:28:43.193621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.193602    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo" pod "b540b40d-ecb8-4938-803c-38a23bf724fd" (UID: "b540b40d-ecb8-4938-803c-38a23bf724fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:28:43.250170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.250146    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"
Nov 03 19:28:43.250346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.250174    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg" event={"ID":"f34c2d52-6f21-4157-944d-9a95f76433a5","Type":"ContainerDied","Data":"c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb"}
Nov 03 19:28:43.250346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.250229    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c607e277d7aaa89f6d2e3d4663d7b10bed35c8bb55edbcc42f2d20542928decb"
Nov 03 19:28:43.252402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.252376    4944 generic.go:356] "Generic (PLEG): container finished" podID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerID="28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a" exitCode=0
Nov 03 19:28:43.252545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.252469    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerDied","Data":"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"}
Nov 03 19:28:43.252545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.252511    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x" event={"ID":"b540b40d-ecb8-4938-803c-38a23bf724fd","Type":"ContainerDied","Data":"ea502e314a7b11e77d382b5a8820d1111ceb5ea890a11c27eedbde2b1f931743"}
Nov 03 19:28:43.252681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.252549    4944 scope.go:122] "RemoveContainer" containerID="28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"
Nov 03 19:28:43.253072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.252715    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"
Nov 03 19:28:43.272166 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.272144    4944 scope.go:122] "RemoveContainer" containerID="7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"
Nov 03 19:28:43.281242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.281210    4944 scope.go:122] "RemoveContainer" containerID="28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"
Nov 03 19:28:43.281520 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:43.281497    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a\": container with ID starting with 28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a not found: ID does not exist" containerID="28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"
Nov 03 19:28:43.281618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.281532    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a"} err="failed to get container status \"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a\": rpc error: code = NotFound desc = could not find container \"28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a\": container with ID starting with 28f154e44cccb64a589912819c3b644ea0fcf6278f32a5272ddb018d6294036a not found: ID does not exist"
Nov 03 19:28:43.281618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.281556    4944 scope.go:122] "RemoveContainer" containerID="7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"
Nov 03 19:28:43.281842 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:28:43.281818    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a\": container with ID starting with 7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a not found: ID does not exist" containerID="7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"
Nov 03 19:28:43.281918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.281851    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a"} err="failed to get container status \"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a\": rpc error: code = NotFound desc = could not find container \"7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a\": container with ID starting with 7afbee85c07f8ccce552272d69ecbf094039f33d8ba4683b1bb0b90956bda15a not found: ID does not exist"
Nov 03 19:28:43.293591 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.293569    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.293675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.293594    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.293675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.293605    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5n6\" (UniqueName: \"kubernetes.io/projected/b540b40d-ecb8-4938-803c-38a23bf724fd-kube-api-access-wr5n6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.293675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.293614    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b540b40d-ecb8-4938-803c-38a23bf724fd-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:28:43.462486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.462394    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 19:28:43.466247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.466221    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-010ed3-openshift-default-54fd45d845-kb66x"]
Nov 03 19:28:43.935580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:43.935544    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" path="/var/lib/kubelet/pods/b540b40d-ecb8-4938-803c-38a23bf724fd/volumes"
Nov 03 19:28:44.454668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:44.454633    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:49.264732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:49.264693    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:53.769471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:53.769434    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:28:58.555730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:28:58.555695    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:02.710782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:02.710741    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:03.465990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.465959    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:03.581217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581180    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"]
Nov 03 19:29:03.581857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581837    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:03.581857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581856    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:03.582065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581911    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f34c2d52-6f21-4157-944d-9a95f76433a5" containerName="revoke-keys"
Nov 03 19:29:03.582065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581920    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34c2d52-6f21-4157-944d-9a95f76433a5" containerName="revoke-keys"
Nov 03 19:29:03.582065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581938    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:03.582065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.581947    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:03.582282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.582075    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:03.582282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.582098    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34c2d52-6f21-4157-944d-9a95f76433a5" containerName="revoke-keys"
Nov 03 19:29:03.585943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.585922    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.622345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.622305    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnwcz\" (UniqueName: \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.622558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.622517    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.622637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.622617    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.681111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.681079    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:29:03.681111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.681102    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:29:03.684270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.684245    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"]
Nov 03 19:29:03.687041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.687025    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:29:03.687119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.687071    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:29:03.713439 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.713410    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:03.724216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.724154    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.724318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.724274    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnwcz\" (UniqueName: \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.724380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.724328    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.724960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.724940    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.726460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.726431    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.781561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.781530    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnwcz\" (UniqueName: \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\") pod \"maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:03.902607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:03.902571    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:04.276263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:04.276226    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"]
Nov 03 19:29:04.282126 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:29:04.282092    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5953eb78_79bc_4e43_800c_77752ef81057.slice/crio-3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a WatchSource:0}: Error finding container 3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a: Status 404 returned error can't find the container with id 3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a
Nov 03 19:29:04.382034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:04.382002    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt" event={"ID":"5953eb78-79bc-4e43-800c-77752ef81057","Type":"ContainerStarted","Data":"3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a"}
Nov 03 19:29:04.763454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:04.763427    4944 scope.go:122] "RemoveContainer" containerID="3abf39b8c21b293c9d6115b85b61e01919c45cd15cf05a84ae09a8159091dd80"
Nov 03 19:29:04.775465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:04.775445    4944 scope.go:122] "RemoveContainer" containerID="ae76adf3bc354be8901d1853119420be935759f19690caa8a1e96b3392750813"
Nov 03 19:29:05.225276 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:29:05.225241    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:29:05.399327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:05.399289    4944 generic.go:356] "Generic (PLEG): container finished" podID="5953eb78-79bc-4e43-800c-77752ef81057" containerID="fd9183d2d7ed9cdeb0dbf6a2a099585ecb469a557e4cb915039a3cef11fa2495" exitCode=0
Nov 03 19:29:05.399513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:05.399385    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt" event={"ID":"5953eb78-79bc-4e43-800c-77752ef81057","Type":"ContainerDied","Data":"fd9183d2d7ed9cdeb0dbf6a2a099585ecb469a557e4cb915039a3cef11fa2495"}
Nov 03 19:29:06.554574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.554546    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:06.656613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.656526    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\") pod \"5953eb78-79bc-4e43-800c-77752ef81057\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") "
Nov 03 19:29:06.656613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.656603    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\") pod \"5953eb78-79bc-4e43-800c-77752ef81057\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") "
Nov 03 19:29:06.656849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.656754    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\" (UniqueName: \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\") pod \"5953eb78-79bc-4e43-800c-77752ef81057\" (UID: \"5953eb78-79bc-4e43-800c-77752ef81057\") "
Nov 03 19:29:06.657009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.656984    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca" pod "5953eb78-79bc-4e43-800c-77752ef81057" (UID: "5953eb78-79bc-4e43-800c-77752ef81057"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:29:06.657258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.657236    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:06.658756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.658724    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca" pod "5953eb78-79bc-4e43-800c-77752ef81057" (UID: "5953eb78-79bc-4e43-800c-77752ef81057"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:29:06.658877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.658835    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz" pod "5953eb78-79bc-4e43-800c-77752ef81057" (UID: "5953eb78-79bc-4e43-800c-77752ef81057"). InnerVolumeSpecName "kube-api-access-qnwcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:29:06.758859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.758828    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5953eb78-79bc-4e43-800c-77752ef81057-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:06.758859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:06.758853    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnwcz\" (UniqueName: \"kubernetes.io/projected/5953eb78-79bc-4e43-800c-77752ef81057-kube-api-access-qnwcz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:07.413665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:07.413637    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"
Nov 03 19:29:07.413876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:07.413632    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt" event={"ID":"5953eb78-79bc-4e43-800c-77752ef81057","Type":"ContainerDied","Data":"3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a"}
Nov 03 19:29:07.413876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:07.413766    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dc4c5b4011abba7feb507d47edfd9de12a81cd10af83078ab505fc411e4687a"
Nov 03 19:29:08.506669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:08.506637    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:13.245132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:13.245053    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:15.223677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:15.223641    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:29:15.689274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:15.689242    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:15.968150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:15.968052    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:17.922016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:17.921974    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:20.224552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:20.224496    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing" containerID="cri-o://704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129" gracePeriod=25
Nov 03 19:29:21.206811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:21.206745    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:29:22.240083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:22.240050    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:22.485085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:22.485045    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:27.034200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:27.034157    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:28.047704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:28.047667    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"]
Nov 03 19:29:28.094103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:28.094071    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"]
Nov 03 19:29:28.096847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:28.096816    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b1fdf6e1-pddmr"]
Nov 03 19:29:28.099156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:28.099134    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc8aec8c-7xxml"]
Nov 03 19:29:29.046717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.046683    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"]
Nov 03 19:29:29.093806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.093761    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54598209-ksrgj"]
Nov 03 19:29:29.097871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.097845    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"]
Nov 03 19:29:29.100158 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.100134    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22f4af47-dv8zg"]
Nov 03 19:29:29.934277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.934242    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b07bdbe-61e3-429e-8b89-08fce8322677" path="/var/lib/kubelet/pods/2b07bdbe-61e3-429e-8b89-08fce8322677/volumes"
Nov 03 19:29:29.934672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.934658    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8616bd-110e-4f79-b251-b4a15802f826" path="/var/lib/kubelet/pods/9c8616bd-110e-4f79-b251-b4a15802f826/volumes"
Nov 03 19:29:29.935065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.935051    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b866af48-68a2-449c-afe5-1e64e05619f4" path="/var/lib/kubelet/pods/b866af48-68a2-449c-afe5-1e64e05619f4/volumes"
Nov 03 19:29:29.935715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:29.935700    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e947238f-4bd7-4953-806f-c59544f1b53b" path="/var/lib/kubelet/pods/e947238f-4bd7-4953-806f-c59544f1b53b/volumes"
Nov 03 19:29:31.206971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:31.206931    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:29:31.608866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:31.608828    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:32.221725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:32.221688    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:32.473160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:32.473062    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:35.225983 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:29:35.225944    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:29:36.413740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:36.413679    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:39.397009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:39.396971    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:40.481605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.481547    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:29:40.555192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.555164    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\" (UniqueName: \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\") pod \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") "
Nov 03 19:29:40.555350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.555262    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\") pod \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\" (UID: \"9deecb35-f8be-4fbb-ba3e-68968fd1ff79\") "
Nov 03 19:29:40.555610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.555586    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume" pod "9deecb35-f8be-4fbb-ba3e-68968fd1ff79" (UID: "9deecb35-f8be-4fbb-ba3e-68968fd1ff79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:29:40.557243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.557223    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd" pod "9deecb35-f8be-4fbb-ba3e-68968fd1ff79" (UID: "9deecb35-f8be-4fbb-ba3e-68968fd1ff79"). InnerVolumeSpecName "kube-api-access-wc4wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:29:40.619275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.619239    4944 generic.go:356] "Generic (PLEG): container finished" podID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerID="704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129" exitCode=0
Nov 03 19:29:40.619451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.619284    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"
Nov 03 19:29:40.619451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.619317    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" event={"ID":"9deecb35-f8be-4fbb-ba3e-68968fd1ff79","Type":"ContainerDied","Data":"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"}
Nov 03 19:29:40.619451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.619378    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj" event={"ID":"9deecb35-f8be-4fbb-ba3e-68968fd1ff79","Type":"ContainerDied","Data":"3234fe021e9da1bb23a08028c85dd26f3aeea9f20969e2dc151c5d549e431d2b"}
Nov 03 19:29:40.619451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.619410    4944 scope.go:122] "RemoveContainer" containerID="704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"
Nov 03 19:29:40.631536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.631518    4944 scope.go:122] "RemoveContainer" containerID="704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"
Nov 03 19:29:40.631916 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:29:40.631884    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129\": container with ID starting with 704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129 not found: ID does not exist" containerID="704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"
Nov 03 19:29:40.632016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.631926    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129"} err="failed to get container status \"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129\": rpc error: code = NotFound desc = could not find container \"704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129\": container with ID starting with 704eb455bad5b38792f903ddc2f44390e6354a7a1798ed7dfacdb2862288c129 not found: ID does not exist"
Nov 03 19:29:40.657296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.657268    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc4wd\" (UniqueName: \"kubernetes.io/projected/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-kube-api-access-wc4wd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:40.657399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.657300    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deecb35-f8be-4fbb-ba3e-68968fd1ff79-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:40.774767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.774725    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:29:40.781305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:40.781270    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1410ed-7766bb9c54-4jrvj"]
Nov 03 19:29:41.021167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:41.021132    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:41.942168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:41.942133    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" path="/var/lib/kubelet/pods/9deecb35-f8be-4fbb-ba3e-68968fd1ff79/volumes"
Nov 03 19:29:42.416943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.416905    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"]
Nov 03 19:29:42.417530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417510    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing"
Nov 03 19:29:42.417653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417532    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing"
Nov 03 19:29:42.417653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417551    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5953eb78-79bc-4e43-800c-77752ef81057" containerName="revoke-keys"
Nov 03 19:29:42.417653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417559    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5953eb78-79bc-4e43-800c-77752ef81057" containerName="revoke-keys"
Nov 03 19:29:42.417856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417683    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b540b40d-ecb8-4938-803c-38a23bf724fd" containerName="istio-proxy"
Nov 03 19:29:42.417856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417701    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9deecb35-f8be-4fbb-ba3e-68968fd1ff79" containerName="payload-processing"
Nov 03 19:29:42.417856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.417715    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5953eb78-79bc-4e43-800c-77752ef81057" containerName="revoke-keys"
Nov 03 19:29:42.422683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.422662    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.475315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.475287    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"]
Nov 03 19:29:42.481392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.481361    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.481600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.481576    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.481681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.481620    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9qb6\" (UniqueName: \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.517950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.517923    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:29:42.517950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.517942    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:29:42.518353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.518336    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:29:42.518660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.518637    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:29:42.583327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.583294    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.583463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.583336    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9qb6\" (UniqueName: \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.583536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.583461    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.584151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.584130    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.585603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.585583    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.641528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.641498    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9qb6\" (UniqueName: \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\") pod \"maas-api-revoke-sub-36b44dc7-gwfkm\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:42.736423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:42.736340    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:43.109056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:43.109022    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"]
Nov 03 19:29:43.114958 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:29:43.114932    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd272c3e4_c581_4594_89ba_0e4593b96fb1.slice/crio-87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae WatchSource:0}: Error finding container 87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae: Status 404 returned error can't find the container with id 87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae
Nov 03 19:29:43.641244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:43.641212    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm" event={"ID":"d272c3e4-c581-4594-89ba-0e4593b96fb1","Type":"ContainerStarted","Data":"07ea36cb063dbb05ab2646bb8b88499904500eaa678d4e2d22c1a9c7940d8af4"}
Nov 03 19:29:43.641419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:43.641271    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm" event={"ID":"d272c3e4-c581-4594-89ba-0e4593b96fb1","Type":"ContainerStarted","Data":"87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae"}
Nov 03 19:29:43.751998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:43.751946    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm" podStartSLOduration=1.751931607 podStartE2EDuration="1.751931607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:43.749600321 +0000 UTC m=+2682.026953917" watchObservedRunningTime="2026-11-03 19:29:43.751931607 +0000 UTC m=+2682.029285189"
Nov 03 19:29:44.649578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:44.649543    4944 generic.go:356] "Generic (PLEG): container finished" podID="d272c3e4-c581-4594-89ba-0e4593b96fb1" containerID="07ea36cb063dbb05ab2646bb8b88499904500eaa678d4e2d22c1a9c7940d8af4" exitCode=0
Nov 03 19:29:44.650021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:44.649628    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm" event={"ID":"d272c3e4-c581-4594-89ba-0e4593b96fb1","Type":"ContainerDied","Data":"07ea36cb063dbb05ab2646bb8b88499904500eaa678d4e2d22c1a9c7940d8af4"}
Nov 03 19:29:45.042017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.041978    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:45.842142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.842119    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:45.926842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.926755    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\") pod \"d272c3e4-c581-4594-89ba-0e4593b96fb1\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") "
Nov 03 19:29:45.926842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.926818    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\") pod \"d272c3e4-c581-4594-89ba-0e4593b96fb1\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") "
Nov 03 19:29:45.927121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.926968    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\" (UniqueName: \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\") pod \"d272c3e4-c581-4594-89ba-0e4593b96fb1\" (UID: \"d272c3e4-c581-4594-89ba-0e4593b96fb1\") "
Nov 03 19:29:45.927212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.927186    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca" pod "d272c3e4-c581-4594-89ba-0e4593b96fb1" (UID: "d272c3e4-c581-4594-89ba-0e4593b96fb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:29:45.927327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.927302    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:45.928949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.928924    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca" pod "d272c3e4-c581-4594-89ba-0e4593b96fb1" (UID: "d272c3e4-c581-4594-89ba-0e4593b96fb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:29:45.929120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:45.929095    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6" pod "d272c3e4-c581-4594-89ba-0e4593b96fb1" (UID: "d272c3e4-c581-4594-89ba-0e4593b96fb1"). InnerVolumeSpecName "kube-api-access-c9qb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:29:46.029052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.029014    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9qb6\" (UniqueName: \"kubernetes.io/projected/d272c3e4-c581-4594-89ba-0e4593b96fb1-kube-api-access-c9qb6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:46.029052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.029048    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d272c3e4-c581-4594-89ba-0e4593b96fb1-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:29:46.653857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.653824    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:46.665091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.665058    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm" event={"ID":"d272c3e4-c581-4594-89ba-0e4593b96fb1","Type":"ContainerDied","Data":"87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae"}
Nov 03 19:29:46.665091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.665082    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"
Nov 03 19:29:46.665268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.665117    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87a0062ccfa479a4c03f43a4f49ef55e62abba68eee328d5cb40cf7c9323cbae"
Nov 03 19:29:46.914041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:46.913956    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:47.149845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:47.149804    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"]
Nov 03 19:29:47.159615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:47.159587    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-efe6c7-gzjd5"]
Nov 03 19:29:47.168727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:47.168641    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:47.663858 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:47.663828    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:47.934199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:47.934121    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103fd114-1b52-4d3a-ab08-aa8c72adf652" path="/var/lib/kubelet/pods/103fd114-1b52-4d3a-ab08-aa8c72adf652/volumes"
Nov 03 19:29:52.169623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:52.169580    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:53.785583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:53.785548    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:54.039016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:54.038909    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:54.284459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:54.284428    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:54.946546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:54.946508    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:55.984396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:55.984360    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:29:55.985017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:55.985000    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d272c3e4-c581-4594-89ba-0e4593b96fb1" containerName="revoke-keys"
Nov 03 19:29:55.985017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:55.985018    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d272c3e4-c581-4594-89ba-0e4593b96fb1" containerName="revoke-keys"
Nov 03 19:29:55.985259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:55.985244    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d272c3e4-c581-4594-89ba-0e4593b96fb1" containerName="revoke-keys"
Nov 03 19:29:55.989295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:55.989270    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.038531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.038438    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fcb646-openshift-default-dockercfg-wncrp\""
Nov 03 19:29:56.088548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.088513    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:29:56.146178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146134    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146190    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146342    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146424    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146489    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146515    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146592    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrh74\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146629    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.146701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.146678    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.190325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.190270    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:56.247478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247451    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrh74\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247617 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247492    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247616    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247706 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247687    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247731    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247837    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.247955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247930    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.247939    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248040    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248058    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248108    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248135    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248181    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.248431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.248392    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.250304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.250283    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.250689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.250670    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.305654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.305618    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.305811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.305759    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrh74\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\") pod \"e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") " pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.463074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.463026    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:56.603103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.603022    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:56.987600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:56.987504    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:29:57.035030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.034996    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:29:57.039427 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:29:57.039391    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf7dd66_530e_41f3_abe3_d888e11e5526.slice/crio-f33189b3204c7e1fb5792e27750cf487d47a5ce3c023488ef5cc6df553b7712d WatchSource:0}: Error finding container f33189b3204c7e1fb5792e27750cf487d47a5ce3c023488ef5cc6df553b7712d: Status 404 returned error can't find the container with id f33189b3204c7e1fb5792e27750cf487d47a5ce3c023488ef5cc6df553b7712d
Nov 03 19:29:57.042061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.042028    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:29:57.042181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.042100    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:29:57.042181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.042130    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:29:57.738144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.738106    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerStarted","Data":"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"}
Nov 03 19:29:57.738366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.738163    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerStarted","Data":"f33189b3204c7e1fb5792e27750cf487d47a5ce3c023488ef5cc6df553b7712d"}
Nov 03 19:29:57.898679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:57.898620    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podStartSLOduration=2.8986055090000002 podStartE2EDuration="2.898605509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:29:57.896493882 +0000 UTC m=+2696.173847465" watchObservedRunningTime="2026-11-03 19:29:57.898605509 +0000 UTC m=+2696.175959092"
Nov 03 19:29:58.603863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:58.603825    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:58.609474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:58.609446    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:58.744351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:58.744315    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:29:58.745400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:29:58.745367    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:30:00.167969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.167920    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:00.172308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.172281    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.175014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.172726    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:00.177108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.177086    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:00.177220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.177202    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.181199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.181179    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.213978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.213951    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:00.217853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.217834    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:00.217975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.217961    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.221704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.221687    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:00.221845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.221830    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.225586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.225570    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:00.225665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.225647    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.233006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.232968    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:00.233120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.233078    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.242795 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.242767    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.257229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.257200    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:30:00.257322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.257228    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:30:00.257322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.257240    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:30:00.265660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.263387    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:00.268313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.268246    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:00.274308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.274066    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:00.278162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.278140    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:00.299348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299320    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hkvd\" (UniqueName: \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.299498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299367    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.299498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299479    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.299630 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299598    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.299713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299697    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-577dq\" (UniqueName: \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.299779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.299766    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.309907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.309885    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:00.327894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.327829    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:00.333269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.333246    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:00.337663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.337633    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:00.400527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400491    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-577dq\" (UniqueName: \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.400715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400534    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.400715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400574    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hkvd\" (UniqueName: \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.400715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400595    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsvlp\" (UniqueName: \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.400715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400617    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.400715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400654    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6bv\" (UniqueName: \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.401088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400820    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.401088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400873    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.401088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400930    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxn2z\" (UniqueName: \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.401088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.400986    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.401088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401031    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.401322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401144    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.401322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401170    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jb8\" (UniqueName: \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.401322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401298    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkvwz\" (UniqueName: \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.401322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401321    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.401551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401350    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.401551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401350    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.401551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401407    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.401884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.401864    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.457897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.457830    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\") pod \"maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.458025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.457922    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hkvd\" (UniqueName: \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\") pod \"maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.458114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.458092    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-577dq\" (UniqueName: \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\") pod \"maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.487253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.487226    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:00.496095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.496071    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:00.501594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.501569    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:00.504766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.504742    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jb8\" (UniqueName: \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.504907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.504834    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkvwz\" (UniqueName: \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.504984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.504897    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.505123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505101    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.505241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505223    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsvlp\" (UniqueName: \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.505316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505275    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6bv\" (UniqueName: \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.505378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505321    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.505378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505367    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxn2z\" (UniqueName: \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.505663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505396    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.505663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505437    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.505663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.505599    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.506310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.506285    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.506415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.506289    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.506711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.506689    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.506952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.506925    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.565659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.565622    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkvwz\" (UniqueName: \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.566549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.566522    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jb8\" (UniqueName: \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\") pod \"maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.569364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.569020    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6bv\" (UniqueName: \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\") pod \"maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.577843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.575406    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsvlp\" (UniqueName: \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.582283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.581445    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:00.597604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.597572    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxn2z\" (UniqueName: \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\") pod \"maas-api-key-cleanup-29895570-8lnt2\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.714652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.714577    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:00.838519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.838485    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:00.847758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.847732    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:00.861043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.860983    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:00.875938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.875700    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:00.885829 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:00.885636    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704e7b63_53b7_4f29_9bde_9840e56dfc06.slice/crio-7f6cbbca4eed2636d6d55d656dda61897947d0e4c5f1f896a14b177b34866ebd WatchSource:0}: Error finding container 7f6cbbca4eed2636d6d55d656dda61897947d0e4c5f1f896a14b177b34866ebd: Status 404 returned error can't find the container with id 7f6cbbca4eed2636d6d55d656dda61897947d0e4c5f1f896a14b177b34866ebd
Nov 03 19:30:00.890895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.890867    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:00.963514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.963479    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:00.969114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.969042    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:00.982496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.982474    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:00.987135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:00.985204    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:00.998682 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:00.998453    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-fb95ae50dc10b157246b766da59f2b98c4cf0afa15c1423b8963610a730bce7a WatchSource:0}: Error finding container fb95ae50dc10b157246b766da59f2b98c4cf0afa15c1423b8963610a730bce7a: Status 404 returned error can't find the container with id fb95ae50dc10b157246b766da59f2b98c4cf0afa15c1423b8963610a730bce7a
Nov 03 19:30:01.219101 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.219074    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:01.225961 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:01.225936    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10d89f7_d332_4aa4_a06a_7c076f01bc31.slice/crio-f652f24969f6eba6df793e8fcd98851eef7ba9a4dec075a320cad422b3186783 WatchSource:0}: Error finding container f652f24969f6eba6df793e8fcd98851eef7ba9a4dec075a320cad422b3186783: Status 404 returned error can't find the container with id f652f24969f6eba6df793e8fcd98851eef7ba9a4dec075a320cad422b3186783
Nov 03 19:30:01.266305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.266282    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:01.271922 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:01.271894    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd9cd22_d452_44bb_93de_16e363d24f99.slice/crio-e5b3b1a7343162843ce50bef1a3ae4d8a7f127398ceba9cdad4825e936a76d28 WatchSource:0}: Error finding container e5b3b1a7343162843ce50bef1a3ae4d8a7f127398ceba9cdad4825e936a76d28: Status 404 returned error can't find the container with id e5b3b1a7343162843ce50bef1a3ae4d8a7f127398ceba9cdad4825e936a76d28
Nov 03 19:30:01.282687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.282662    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:01.286390 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:01.286372    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317bcf7d_984b_4eb5_b584_5a6c9c7dbe10.slice/crio-fa0fe28694bc24846f757a4fdc00eff8078f4d494ffcfc3fd8106fe200317dcc WatchSource:0}: Error finding container fa0fe28694bc24846f757a4fdc00eff8078f4d494ffcfc3fd8106fe200317dcc: Status 404 returned error can't find the container with id fa0fe28694bc24846f757a4fdc00eff8078f4d494ffcfc3fd8106fe200317dcc
Nov 03 19:30:01.364179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.364150    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:01.369405 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:01.369380    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23977d6_9107_4d07_9b2b_34d9c9d49ff8.slice/crio-d34a858a63203c38937f0cb7d2c1dca3f0543a5bd017507801e2fb83c9b2f8cd WatchSource:0}: Error finding container d34a858a63203c38937f0cb7d2c1dca3f0543a5bd017507801e2fb83c9b2f8cd: Status 404 returned error can't find the container with id d34a858a63203c38937f0cb7d2c1dca3f0543a5bd017507801e2fb83c9b2f8cd
Nov 03 19:30:01.752249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.752221    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:01.765518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.765485    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerStarted","Data":"aa349c2b2732ad531a13036ea61d8f81532cc9ab0d2afbe8359cd619d43c3346"}
Nov 03 19:30:01.765669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.765565    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerStarted","Data":"d34a858a63203c38937f0cb7d2c1dca3f0543a5bd017507801e2fb83c9b2f8cd"}
Nov 03 19:30:01.767225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.767199    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerStarted","Data":"1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a"}
Nov 03 19:30:01.767357 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.767247    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerStarted","Data":"fb95ae50dc10b157246b766da59f2b98c4cf0afa15c1423b8963610a730bce7a"}
Nov 03 19:30:01.768731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.768697    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerStarted","Data":"4f72458d068d7cfcd5f31b1826de4e35a689c67104e4c25fe3598a10ccb3d581"}
Nov 03 19:30:01.768862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.768750    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerStarted","Data":"d34a9141a92ead7a7b61448edebc4c8b09463f2bca322af04d60022706cff289"}
Nov 03 19:30:01.770107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.770086    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerStarted","Data":"033b4c59e71b5e03bd5725f025d3be525468dd65401a033932d3c38052425903"}
Nov 03 19:30:01.770206 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.770124    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerStarted","Data":"7f6cbbca4eed2636d6d55d656dda61897947d0e4c5f1f896a14b177b34866ebd"}
Nov 03 19:30:01.771533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.771512    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerStarted","Data":"c95618100b3c2344e0e4cd660976c4c5be61b0c83325d277e129ca2fe118e470"}
Nov 03 19:30:01.771615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.771546    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerStarted","Data":"fa0fe28694bc24846f757a4fdc00eff8078f4d494ffcfc3fd8106fe200317dcc"}
Nov 03 19:30:01.772991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.772964    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerStarted","Data":"46096d37313ec087abaa9d0b387bdb84769b9e2baad657e708c3ef6d9792d2bf"}
Nov 03 19:30:01.773093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.773015    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerStarted","Data":"dc0b052bfa1a903b8929dcff1cf1a61479e4bdeea9ccc59779651f708c741d87"}
Nov 03 19:30:01.774036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.774016    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerStarted","Data":"e5b3b1a7343162843ce50bef1a3ae4d8a7f127398ceba9cdad4825e936a76d28"}
Nov 03 19:30:01.775324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.775301    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerStarted","Data":"eb21fdb4579d4e849b9ca0f350594d496da8c7d067020145b0a3189cccb259a3"}
Nov 03 19:30:01.775419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.775344    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerStarted","Data":"f652f24969f6eba6df793e8fcd98851eef7ba9a4dec075a320cad422b3186783"}
Nov 03 19:30:01.876384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.876324    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" podStartSLOduration=1.876310736 podStartE2EDuration="1.876310736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:01.873168828 +0000 UTC m=+2700.150522413" watchObservedRunningTime="2026-11-03 19:30:01.876310736 +0000 UTC m=+2700.153664318"
Nov 03 19:30:01.988748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:01.988686    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" podStartSLOduration=1.988671737 podStartE2EDuration="1.988671737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:01.986525859 +0000 UTC m=+2700.263879457" watchObservedRunningTime="2026-11-03 19:30:01.988671737 +0000 UTC m=+2700.266025319"
Nov 03 19:30:02.096304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.096198    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" podStartSLOduration=2.096184938 podStartE2EDuration="2.096184938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.09464498 +0000 UTC m=+2700.371998564" watchObservedRunningTime="2026-11-03 19:30:02.096184938 +0000 UTC m=+2700.373538521"
Nov 03 19:30:02.204452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.204399    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" podStartSLOduration=2.204384594 podStartE2EDuration="2.204384594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.201980753 +0000 UTC m=+2700.479334335" watchObservedRunningTime="2026-11-03 19:30:02.204384594 +0000 UTC m=+2700.481738198"
Nov 03 19:30:02.354862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.354769    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:30:02.357150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.357128    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:30:02.423897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.423848    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" podStartSLOduration=2.423835428 podStartE2EDuration="2.423835428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.314052607 +0000 UTC m=+2700.591406191" watchObservedRunningTime="2026-11-03 19:30:02.423835428 +0000 UTC m=+2700.701189011"
Nov 03 19:30:02.424678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.424644    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" podStartSLOduration=2.424634092 podStartE2EDuration="2.424634092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.421431521 +0000 UTC m=+2700.698785105" watchObservedRunningTime="2026-11-03 19:30:02.424634092 +0000 UTC m=+2700.701987674"
Nov 03 19:30:02.529553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.529498    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" podStartSLOduration=2.529481546 podStartE2EDuration="2.529481546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:02.528938575 +0000 UTC m=+2700.806292157" watchObservedRunningTime="2026-11-03 19:30:02.529481546 +0000 UTC m=+2700.806835129"
Nov 03 19:30:02.783174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.783135    4944 generic.go:356] "Generic (PLEG): container finished" podID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerID="5dc0f710256387b0db886e8b8c8c9dfcf8dd1e0733002c61d3fb3f3544b694b1" exitCode=60
Nov 03 19:30:02.783363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.783216    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerDied","Data":"5dc0f710256387b0db886e8b8c8c9dfcf8dd1e0733002c61d3fb3f3544b694b1"}
Nov 03 19:30:02.783481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.783465    4944 scope.go:122] "RemoveContainer" containerID="5dc0f710256387b0db886e8b8c8c9dfcf8dd1e0733002c61d3fb3f3544b694b1"
Nov 03 19:30:02.787426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.787404    4944 generic.go:356] "Generic (PLEG): container finished" podID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerID="eb21fdb4579d4e849b9ca0f350594d496da8c7d067020145b0a3189cccb259a3" exitCode=60
Nov 03 19:30:02.787767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.787668    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerDied","Data":"eb21fdb4579d4e849b9ca0f350594d496da8c7d067020145b0a3189cccb259a3"}
Nov 03 19:30:02.788074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.788047    4944 scope.go:122] "RemoveContainer" containerID="eb21fdb4579d4e849b9ca0f350594d496da8c7d067020145b0a3189cccb259a3"
Nov 03 19:30:02.790780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.790751    4944 generic.go:356] "Generic (PLEG): container finished" podID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerID="aa349c2b2732ad531a13036ea61d8f81532cc9ab0d2afbe8359cd619d43c3346" exitCode=60
Nov 03 19:30:02.790895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.790819    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerDied","Data":"aa349c2b2732ad531a13036ea61d8f81532cc9ab0d2afbe8359cd619d43c3346"}
Nov 03 19:30:02.791074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.791055    4944 scope.go:122] "RemoveContainer" containerID="aa349c2b2732ad531a13036ea61d8f81532cc9ab0d2afbe8359cd619d43c3346"
Nov 03 19:30:02.792444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.792423    4944 generic.go:356] "Generic (PLEG): container finished" podID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerID="1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a" exitCode=60
Nov 03 19:30:02.792564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.792512    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerDied","Data":"1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a"}
Nov 03 19:30:02.792714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.792693    4944 scope.go:122] "RemoveContainer" containerID="1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a"
Nov 03 19:30:02.794124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.794107    4944 generic.go:356] "Generic (PLEG): container finished" podID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerID="4f72458d068d7cfcd5f31b1826de4e35a689c67104e4c25fe3598a10ccb3d581" exitCode=60
Nov 03 19:30:02.794222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.794200    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerDied","Data":"4f72458d068d7cfcd5f31b1826de4e35a689c67104e4c25fe3598a10ccb3d581"}
Nov 03 19:30:02.794372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.794360    4944 scope.go:122] "RemoveContainer" containerID="4f72458d068d7cfcd5f31b1826de4e35a689c67104e4c25fe3598a10ccb3d581"
Nov 03 19:30:02.795747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.795727    4944 generic.go:356] "Generic (PLEG): container finished" podID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerID="033b4c59e71b5e03bd5725f025d3be525468dd65401a033932d3c38052425903" exitCode=60
Nov 03 19:30:02.795868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.795826    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerDied","Data":"033b4c59e71b5e03bd5725f025d3be525468dd65401a033932d3c38052425903"}
Nov 03 19:30:02.796006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.795992    4944 scope.go:122] "RemoveContainer" containerID="033b4c59e71b5e03bd5725f025d3be525468dd65401a033932d3c38052425903"
Nov 03 19:30:02.798957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.798934    4944 generic.go:356] "Generic (PLEG): container finished" podID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerID="c95618100b3c2344e0e4cd660976c4c5be61b0c83325d277e129ca2fe118e470" exitCode=60
Nov 03 19:30:02.799049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.799021    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerDied","Data":"c95618100b3c2344e0e4cd660976c4c5be61b0c83325d277e129ca2fe118e470"}
Nov 03 19:30:02.799307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.799288    4944 scope.go:122] "RemoveContainer" containerID="c95618100b3c2344e0e4cd660976c4c5be61b0c83325d277e129ca2fe118e470"
Nov 03 19:30:02.800911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.800882    4944 generic.go:356] "Generic (PLEG): container finished" podID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerID="46096d37313ec087abaa9d0b387bdb84769b9e2baad657e708c3ef6d9792d2bf" exitCode=60
Nov 03 19:30:02.801007 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.800907    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerDied","Data":"46096d37313ec087abaa9d0b387bdb84769b9e2baad657e708c3ef6d9792d2bf"}
Nov 03 19:30:02.801153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:02.801136    4944 scope.go:122] "RemoveContainer" containerID="46096d37313ec087abaa9d0b387bdb84769b9e2baad657e708c3ef6d9792d2bf"
Nov 03 19:30:03.566757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.566720    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:03.810355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.810177    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerStarted","Data":"a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"}
Nov 03 19:30:03.814381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.813952    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerStarted","Data":"3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"}
Nov 03 19:30:03.816393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.816343    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerStarted","Data":"a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"}
Nov 03 19:30:03.818881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.818850    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerStarted","Data":"c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14"}
Nov 03 19:30:03.821444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.821415    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerStarted","Data":"f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"}
Nov 03 19:30:03.828659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.828633    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerStarted","Data":"2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"}
Nov 03 19:30:03.832426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.832390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerStarted","Data":"4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"}
Nov 03 19:30:03.850844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:03.835217    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerStarted","Data":"2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"}
Nov 03 19:30:04.027295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.027248    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" podStartSLOduration=4.02723303 podStartE2EDuration="4.02723303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:04.025270183 +0000 UTC m=+2702.302623765" watchObservedRunningTime="2026-11-03 19:30:04.02723303 +0000 UTC m=+2702.304586652"
Nov 03 19:30:04.454055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.453973    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:04.703415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.703379    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:04.750510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.750481    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:30:04.754616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.754598    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:04.808332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.808301    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fcb646\""
Nov 03 19:30:04.813239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.813213    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:30:04.814090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.814066    4944 scope.go:122] "RemoveContainer" containerID="172b0d9e8626b7a28bfece447e2bc9e8b7c53b2bd72efa442cfc7edfc284938a"
Nov 03 19:30:04.825695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.825672    4944 scope.go:122] "RemoveContainer" containerID="ebcd7f507f91249e8456cf4b40716dd044d1cf4694a4c77cae4a1aa56e36a7d9"
Nov 03 19:30:04.898190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.898052    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:04.902456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.902419    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8w4d\" (UniqueName: \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:04.904078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.898749    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fcb646-dockercfg-h7m28\""
Nov 03 19:30:04.914741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.914712    4944 generic.go:356] "Generic (PLEG): container finished" podID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerID="a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90" exitCode=60
Nov 03 19:30:04.915174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.915148    4944 scope.go:122] "RemoveContainer" containerID="a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"
Nov 03 19:30:04.915289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.915200    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerDied","Data":"a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"}
Nov 03 19:30:04.915289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.915260    4944 scope.go:122] "RemoveContainer" containerID="033b4c59e71b5e03bd5725f025d3be525468dd65401a033932d3c38052425903"
Nov 03 19:30:04.915519 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.915487    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b_odh-ai-gateway-infra(704e7b63-53b7-4f29-9bde-9840e56dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06"
Nov 03 19:30:04.917192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.917168    4944 generic.go:356] "Generic (PLEG): container finished" podID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerID="c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14" exitCode=60
Nov 03 19:30:04.917288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.917254    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerDied","Data":"c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14"}
Nov 03 19:30:04.917564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.917545    4944 scope.go:122] "RemoveContainer" containerID="c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14"
Nov 03 19:30:04.917870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.917846    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:30:04.917947 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.917906    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr_odh-ai-gateway-infra(317bcf7d-984b-4eb5-b584-5a6c9c7dbe10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10"
Nov 03 19:30:04.921984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.921965    4944 generic.go:356] "Generic (PLEG): container finished" podID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerID="f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70" exitCode=60
Nov 03 19:30:04.922329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.922313    4944 scope.go:122] "RemoveContainer" containerID="f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"
Nov 03 19:30:04.922625 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.922589    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg_odh-ai-gateway-infra(fef8bffc-dbb4-44cb-be80-36f2d461dbf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5"
Nov 03 19:30:04.923963 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.923939    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerDied","Data":"f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"}
Nov 03 19:30:04.924078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.924039    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:04.924146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.924115    4944 generic.go:356] "Generic (PLEG): container finished" podID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerID="2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16" exitCode=60
Nov 03 19:30:04.924146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.924132    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerDied","Data":"2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"}
Nov 03 19:30:04.924399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.924384    4944 scope.go:122] "RemoveContainer" containerID="2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"
Nov 03 19:30:04.924952 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.924640    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp_odh-ai-gateway-infra(0fd9cd22-d452-44bb-93de-16e363d24f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99"
Nov 03 19:30:04.927425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.927404    4944 generic.go:356] "Generic (PLEG): container finished" podID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerID="4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a" exitCode=60
Nov 03 19:30:04.927531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.927503    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerDied","Data":"4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"}
Nov 03 19:30:04.927766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.927749    4944 scope.go:122] "RemoveContainer" containerID="4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"
Nov 03 19:30:04.928001 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.927975    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg_odh-ai-gateway-infra(b10d89f7-d332-4aa4-a06a-7c076f01bc31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31"
Nov 03 19:30:04.931541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.931509    4944 generic.go:356] "Generic (PLEG): container finished" podID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerID="2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca" exitCode=60
Nov 03 19:30:04.931701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.931681    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerDied","Data":"2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"}
Nov 03 19:30:04.932104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.932082    4944 scope.go:122] "RemoveContainer" containerID="2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"
Nov 03 19:30:04.932553 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.932525    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-8lnt2_odh-ai-gateway-infra(a23977d6-9107-4d07-9b2b-34d9c9d49ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8"
Nov 03 19:30:04.935412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.935390    4944 generic.go:356] "Generic (PLEG): container finished" podID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerID="a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e" exitCode=60
Nov 03 19:30:04.935515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.935497    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerDied","Data":"a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"}
Nov 03 19:30:04.935935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.935911    4944 scope.go:122] "RemoveContainer" containerID="a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"
Nov 03 19:30:04.936286 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.936263    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7_odh-ai-gateway-infra(1a8fe5ec-7703-4034-b56a-11a061c658e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2"
Nov 03 19:30:04.938363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.938338    4944 generic.go:356] "Generic (PLEG): container finished" podID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerID="3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1" exitCode=60
Nov 03 19:30:04.938466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.938447    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerDied","Data":"3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"}
Nov 03 19:30:04.939438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.939287    4944 scope.go:122] "RemoveContainer" containerID="3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"
Nov 03 19:30:04.939702 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:04.939644    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv_odh-ai-gateway-infra(3955a2c1-b2f8-4365-b32e-085ee6a771a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8"
Nov 03 19:30:04.940144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.940125    4944 scope.go:122] "RemoveContainer" containerID="c95618100b3c2344e0e4cd660976c4c5be61b0c83325d277e129ca2fe118e470"
Nov 03 19:30:04.954490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.954467    4944 scope.go:122] "RemoveContainer" containerID="46096d37313ec087abaa9d0b387bdb84769b9e2baad657e708c3ef6d9792d2bf"
Nov 03 19:30:04.967477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.967452    4944 scope.go:122] "RemoveContainer" containerID="5dc0f710256387b0db886e8b8c8c9dfcf8dd1e0733002c61d3fb3f3544b694b1"
Nov 03 19:30:04.975169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.975145    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:30:04.986902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.986876    4944 scope.go:122] "RemoveContainer" containerID="eb21fdb4579d4e849b9ca0f350594d496da8c7d067020145b0a3189cccb259a3"
Nov 03 19:30:04.999903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:04.999880    4944 scope.go:122] "RemoveContainer" containerID="aa349c2b2732ad531a13036ea61d8f81532cc9ab0d2afbe8359cd619d43c3346"
Nov 03 19:30:05.004613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.004581    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zphsq\" (UniqueName: \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.004727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.004625    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.004799 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.004742    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8w4d\" (UniqueName: \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.005319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.005293    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.005556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.005533    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.015130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.015109    4944 scope.go:122] "RemoveContainer" containerID="1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a"
Nov 03 19:30:05.028635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.028617    4944 scope.go:122] "RemoveContainer" containerID="4f72458d068d7cfcd5f31b1826de4e35a689c67104e4c25fe3598a10ccb3d581"
Nov 03 19:30:05.061884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.061856    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8w4d\" (UniqueName: \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\") pod \"payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.067748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.067726    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.107328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.107297    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.107508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.107418    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zphsq\" (UniqueName: \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.108198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.108170    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.177224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.177190    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zphsq\" (UniqueName: \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\") pod \"payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.224742 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.224709    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:30:05.250156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.250127    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.445835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.445777    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:30:05.451262 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:05.451233    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd134d9_e71f_43e1_910a_d37a78a0a49f.slice/crio-fea310309567012333f64a2e7993cdb4d2562a67917f256209de02030ba4456b WatchSource:0}: Error finding container fea310309567012333f64a2e7993cdb4d2562a67917f256209de02030ba4456b: Status 404 returned error can't find the container with id fea310309567012333f64a2e7993cdb4d2562a67917f256209de02030ba4456b
Nov 03 19:30:05.622379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.622349    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:30:05.627596 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:05.627563    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d8d9e4_15e8_46da_9b31_39821a8988a2.slice/crio-36dbb1c6f2a8d5e34ac400fafec48b7cf167761594a9682965896e28f2574ebf WatchSource:0}: Error finding container 36dbb1c6f2a8d5e34ac400fafec48b7cf167761594a9682965896e28f2574ebf: Status 404 returned error can't find the container with id 36dbb1c6f2a8d5e34ac400fafec48b7cf167761594a9682965896e28f2574ebf
Nov 03 19:30:05.948853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.948762    4944 scope.go:122] "RemoveContainer" containerID="2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"
Nov 03 19:30:05.949261 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.949087    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp_odh-ai-gateway-infra(0fd9cd22-d452-44bb-93de-16e363d24f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99"
Nov 03 19:30:05.950380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.950359    4944 scope.go:122] "RemoveContainer" containerID="4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"
Nov 03 19:30:05.950659 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.950636    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg_odh-ai-gateway-infra(b10d89f7-d332-4aa4-a06a-7c076f01bc31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31"
Nov 03 19:30:05.951928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.951909    4944 scope.go:122] "RemoveContainer" containerID="2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"
Nov 03 19:30:05.952106 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.952085    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-8lnt2_odh-ai-gateway-infra(a23977d6-9107-4d07-9b2b-34d9c9d49ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8"
Nov 03 19:30:05.953479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.953458    4944 scope.go:122] "RemoveContainer" containerID="a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"
Nov 03 19:30:05.953678 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.953654    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7_odh-ai-gateway-infra(1a8fe5ec-7703-4034-b56a-11a061c658e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2"
Nov 03 19:30:05.954674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.954651    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" event={"ID":"dfd134d9-e71f-43e1-910a-d37a78a0a49f","Type":"ContainerStarted","Data":"1f63272dfae63ad534fda8c1eefcf9039314828038a415a6a093b9e985c2552b"}
Nov 03 19:30:05.954747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.954703    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:05.954747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.954727    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" event={"ID":"dfd134d9-e71f-43e1-910a-d37a78a0a49f","Type":"ContainerStarted","Data":"fea310309567012333f64a2e7993cdb4d2562a67917f256209de02030ba4456b"}
Nov 03 19:30:05.956353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.956334    4944 scope.go:122] "RemoveContainer" containerID="3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"
Nov 03 19:30:05.956545 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.956525    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv_odh-ai-gateway-infra(3955a2c1-b2f8-4365-b32e-085ee6a771a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8"
Nov 03 19:30:05.957938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.957916    4944 scope.go:122] "RemoveContainer" containerID="a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"
Nov 03 19:30:05.958131 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.958111    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b_odh-ai-gateway-infra(704e7b63-53b7-4f29-9bde-9840e56dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06"
Nov 03 19:30:05.959542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.959523    4944 scope.go:122] "RemoveContainer" containerID="c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14"
Nov 03 19:30:05.959730 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.959707    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr_odh-ai-gateway-infra(317bcf7d-984b-4eb5-b584-5a6c9c7dbe10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10"
Nov 03 19:30:05.960829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.960810    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" event={"ID":"16d8d9e4-15e8-46da-9b31-39821a8988a2","Type":"ContainerStarted","Data":"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"}
Nov 03 19:30:05.960910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.960847    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" event={"ID":"16d8d9e4-15e8-46da-9b31-39821a8988a2","Type":"ContainerStarted","Data":"36dbb1c6f2a8d5e34ac400fafec48b7cf167761594a9682965896e28f2574ebf"}
Nov 03 19:30:05.960910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.960866    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:05.962665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:05.962646    4944 scope.go:122] "RemoveContainer" containerID="f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"
Nov 03 19:30:05.962864 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:05.962845    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg_odh-ai-gateway-infra(fef8bffc-dbb4-44cb-be80-36f2d461dbf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5"
Nov 03 19:30:06.384581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:06.384528    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" podStartSLOduration=2.384513847 podStartE2EDuration="2.384513847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:06.381067269 +0000 UTC m=+2704.658420853" watchObservedRunningTime="2026-11-03 19:30:06.384513847 +0000 UTC m=+2704.661867424"
Nov 03 19:30:06.549572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:06.549539    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:06.597691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:06.597633    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" podStartSLOduration=2.597618324 podStartE2EDuration="2.597618324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:06.596140887 +0000 UTC m=+2704.873494469" watchObservedRunningTime="2026-11-03 19:30:06.597618324 +0000 UTC m=+2704.874971907"
Nov 03 19:30:07.000781 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:07.000746    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:08.596992 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:08.596960    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:30:08.715888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:08.715855    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:09.524043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.524003    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"]
Nov 03 19:30:09.529164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.529143    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.564852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.564825    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fzr\" (UniqueName: \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.564977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.564860    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.565071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.565050    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.579709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.579688    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:30:09.595663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.595636    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"]
Nov 03 19:30:09.666243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.666213    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.666707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.666320    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fzr\" (UniqueName: \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.666707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.666354    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.666891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.666872    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.668742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.668721    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.723388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.723358    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fzr\" (UniqueName: \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\") pod \"maas-api-revoke-sub-53927318-m77xr\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:09.843927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:09.843850    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:10.220391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:10.220297    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"]
Nov 03 19:30:10.225527 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:10.225495    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a19da2f_0cda_4463_9856_e8be2dd69c44.slice/crio-56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489 WatchSource:0}: Error finding container 56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489: Status 404 returned error can't find the container with id 56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489
Nov 03 19:30:11.005907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:11.005861    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr" event={"ID":"2a19da2f-0cda-4463-9856-e8be2dd69c44","Type":"ContainerStarted","Data":"dc9ec7aff630ee8cb9beded49b46b31464911103d58577fe507cd90cbb9524aa"}
Nov 03 19:30:11.006368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:11.005946    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr" event={"ID":"2a19da2f-0cda-4463-9856-e8be2dd69c44","Type":"ContainerStarted","Data":"56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489"}
Nov 03 19:30:11.122820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:11.119477    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr" podStartSLOduration=2.1194565 podStartE2EDuration="2.1194565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:11.117186018 +0000 UTC m=+2709.394539614" watchObservedRunningTime="2026-11-03 19:30:11.1194565 +0000 UTC m=+2709.396810086"
Nov 03 19:30:11.155512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:11.155475    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:11.718485 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:11.718453    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:12.013204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:12.013172    4944 generic.go:356] "Generic (PLEG): container finished" podID="2a19da2f-0cda-4463-9856-e8be2dd69c44" containerID="dc9ec7aff630ee8cb9beded49b46b31464911103d58577fe507cd90cbb9524aa" exitCode=0
Nov 03 19:30:12.013651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:12.013263    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr" event={"ID":"2a19da2f-0cda-4463-9856-e8be2dd69c44","Type":"ContainerDied","Data":"dc9ec7aff630ee8cb9beded49b46b31464911103d58577fe507cd90cbb9524aa"}
Nov 03 19:30:13.164265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.164233    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:13.310533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.310449    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\") pod \"2a19da2f-0cda-4463-9856-e8be2dd69c44\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") "
Nov 03 19:30:13.310533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.310524    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\" (UniqueName: \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\") pod \"2a19da2f-0cda-4463-9856-e8be2dd69c44\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") "
Nov 03 19:30:13.310726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.310666    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\") pod \"2a19da2f-0cda-4463-9856-e8be2dd69c44\" (UID: \"2a19da2f-0cda-4463-9856-e8be2dd69c44\") "
Nov 03 19:30:13.311080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.311046    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca" pod "2a19da2f-0cda-4463-9856-e8be2dd69c44" (UID: "2a19da2f-0cda-4463-9856-e8be2dd69c44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:13.311325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.311306    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:13.312636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.312614    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca" pod "2a19da2f-0cda-4463-9856-e8be2dd69c44" (UID: "2a19da2f-0cda-4463-9856-e8be2dd69c44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:30:13.312732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.312677    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr" pod "2a19da2f-0cda-4463-9856-e8be2dd69c44" (UID: "2a19da2f-0cda-4463-9856-e8be2dd69c44"). InnerVolumeSpecName "kube-api-access-j9fzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:13.412450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.412408    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a19da2f-0cda-4463-9856-e8be2dd69c44-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:13.412450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:13.412443    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fzr\" (UniqueName: \"kubernetes.io/projected/2a19da2f-0cda-4463-9856-e8be2dd69c44-kube-api-access-j9fzr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:14.028129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.028098    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"
Nov 03 19:30:14.028305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.028126    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr" event={"ID":"2a19da2f-0cda-4463-9856-e8be2dd69c44","Type":"ContainerDied","Data":"56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489"}
Nov 03 19:30:14.028305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.028177    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56a0620c7ceb3284492703d3099c71bd735edcb71c083bd49f1eb4f6e768b489"
Nov 03 19:30:14.927352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.927317    4944 scope.go:122] "RemoveContainer" containerID="4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"
Nov 03 19:30:14.927881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.927743    4944 scope.go:122] "RemoveContainer" containerID="f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"
Nov 03 19:30:14.927946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:14.927918    4944 scope.go:122] "RemoveContainer" containerID="2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"
Nov 03 19:30:15.146131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.146097    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:15.184241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.184154    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:15.343422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.343396    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:15.443345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.443279    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\" (UniqueName: \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\") pod \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") "
Nov 03 19:30:15.443462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.443396    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\") pod \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\" (UID: \"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10\") "
Nov 03 19:30:15.443722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.443697    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca" pod "317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" (UID: "317bcf7d-984b-4eb5-b584-5a6c9c7dbe10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:15.443925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.443909    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:15.445329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.445310    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8" pod "317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" (UID: "317bcf7d-984b-4eb5-b584-5a6c9c7dbe10"). InnerVolumeSpecName "kube-api-access-p2jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:15.545757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.545724    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2jb8\" (UniqueName: \"kubernetes.io/projected/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10-kube-api-access-p2jb8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:15.939666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.939639    4944 scope.go:122] "RemoveContainer" containerID="a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"
Nov 03 19:30:15.940308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:15.940291    4944 scope.go:122] "RemoveContainer" containerID="2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"
Nov 03 19:30:16.060000 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.059973    4944 generic.go:356] "Generic (PLEG): container finished" podID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerID="72826b1321eb2ad0c21617d426af0970de0a42423f24d04cfd4c2b9220b44d3b" exitCode=60
Nov 03 19:30:16.060148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.060045    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerDied","Data":"72826b1321eb2ad0c21617d426af0970de0a42423f24d04cfd4c2b9220b44d3b"}
Nov 03 19:30:16.060148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.060112    4944 scope.go:122] "RemoveContainer" containerID="4de8fb8b5c8f117bdfe3db373ad4aecb1223c09d323e75c95d4f1ea7ee5af45a"
Nov 03 19:30:16.060509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.060489    4944 scope.go:122] "RemoveContainer" containerID="72826b1321eb2ad0c21617d426af0970de0a42423f24d04cfd4c2b9220b44d3b"
Nov 03 19:30:16.060801 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:16.060772    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg_odh-ai-gateway-infra(b10d89f7-d332-4aa4-a06a-7c076f01bc31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31"
Nov 03 19:30:16.062118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.062099    4944 generic.go:356] "Generic (PLEG): container finished" podID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerID="2143c450db60050e7f9888536b01b568fe6525e6fcb5ab98e24fa6c617ae3d03" exitCode=60
Nov 03 19:30:16.062198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.062182    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerDied","Data":"2143c450db60050e7f9888536b01b568fe6525e6fcb5ab98e24fa6c617ae3d03"}
Nov 03 19:30:16.062540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.062524    4944 scope.go:122] "RemoveContainer" containerID="2143c450db60050e7f9888536b01b568fe6525e6fcb5ab98e24fa6c617ae3d03"
Nov 03 19:30:16.062953 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:16.062928    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895570-8lnt2_odh-ai-gateway-infra(a23977d6-9107-4d07-9b2b-34d9c9d49ff8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8"
Nov 03 19:30:16.063479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.063457    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr" event={"ID":"317bcf7d-984b-4eb5-b584-5a6c9c7dbe10","Type":"ContainerDied","Data":"fa0fe28694bc24846f757a4fdc00eff8078f4d494ffcfc3fd8106fe200317dcc"}
Nov 03 19:30:16.063579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.063492    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"
Nov 03 19:30:16.065461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.065442    4944 generic.go:356] "Generic (PLEG): container finished" podID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerID="292627a228d07c0e58c92a09cbff374820c46c42af98f650f26c6d191d7a4ff6" exitCode=60
Nov 03 19:30:16.065547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.065466    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerDied","Data":"292627a228d07c0e58c92a09cbff374820c46c42af98f650f26c6d191d7a4ff6"}
Nov 03 19:30:16.065856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.065839    4944 scope.go:122] "RemoveContainer" containerID="292627a228d07c0e58c92a09cbff374820c46c42af98f650f26c6d191d7a4ff6"
Nov 03 19:30:16.066120 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:16.066099    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg_odh-ai-gateway-infra(fef8bffc-dbb4-44cb-be80-36f2d461dbf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5"
Nov 03 19:30:16.094616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.094597    4944 scope.go:122] "RemoveContainer" containerID="2b5cd052d2713781bc49899d11244bd9d3f50f05ab1043c3ff231d70d5828cca"
Nov 03 19:30:16.116119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.116102    4944 scope.go:122] "RemoveContainer" containerID="c07d3f1fa0327b7392804c6c69030b123021537647d5bf5040bd3ec5bd40fd14"
Nov 03 19:30:16.125197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.125180    4944 scope.go:122] "RemoveContainer" containerID="f581076ca2ee1f59882bc8fbfa8323e63b141f33bfe6f3929ef24129b20c7a70"
Nov 03 19:30:16.488527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.488458    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:16.491597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.491574    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1410ed-29895570-8m4rr"]
Nov 03 19:30:16.689644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.689615    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:16.927818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.927761    4944 scope.go:122] "RemoveContainer" containerID="a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"
Nov 03 19:30:16.961662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.961632    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:16.973057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.973037    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:30:16.973187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:16.973167    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:30:17.071168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.071138    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"]
Nov 03 19:30:17.074884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.074851    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1410ed-dqltt"]
Nov 03 19:30:17.075734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.075714    4944 generic.go:356] "Generic (PLEG): container finished" podID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerID="ae93c58bf223d4daabbc3035d39d38f5ee7a724602f87495afe1cfc38883c2f3" exitCode=60
Nov 03 19:30:17.075844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.075781    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerDied","Data":"ae93c58bf223d4daabbc3035d39d38f5ee7a724602f87495afe1cfc38883c2f3"}
Nov 03 19:30:17.075897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.075883    4944 scope.go:122] "RemoveContainer" containerID="a4797ff644b91eac2900c1f1e5c44833d93d53f6ea1c917bd8d7430a2f280e90"
Nov 03 19:30:17.076243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.076226    4944 scope.go:122] "RemoveContainer" containerID="ae93c58bf223d4daabbc3035d39d38f5ee7a724602f87495afe1cfc38883c2f3"
Nov 03 19:30:17.076543 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:17.076515    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b_odh-ai-gateway-infra(704e7b63-53b7-4f29-9bde-9840e56dfc06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06"
Nov 03 19:30:17.080326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.080304    4944 generic.go:356] "Generic (PLEG): container finished" podID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerID="19b442026cf09836a0da878c20486947b2f864292fde331949e02d4fce7266ff" exitCode=60
Nov 03 19:30:17.080421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.080368    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerDied","Data":"19b442026cf09836a0da878c20486947b2f864292fde331949e02d4fce7266ff"}
Nov 03 19:30:17.080751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.080735    4944 scope.go:122] "RemoveContainer" containerID="19b442026cf09836a0da878c20486947b2f864292fde331949e02d4fce7266ff"
Nov 03 19:30:17.081082 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:17.081052    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp_odh-ai-gateway-infra(0fd9cd22-d452-44bb-93de-16e363d24f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99"
Nov 03 19:30:17.106095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.106071    4944 scope.go:122] "RemoveContainer" containerID="2f09d3835c791b8c5ccd4b8bd325f38a1c5bdb75bb79d37b710c89ecf438bf16"
Nov 03 19:30:17.179933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.179858    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:17.226050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.226017    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:17.287155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.286814    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:17.358263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.358227    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:17.410109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.410059    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:17.447855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.447746    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:17.468775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.468736    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\" (UniqueName: \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\") pod \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") "
Nov 03 19:30:17.468775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.468777    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\") pod \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\" (UID: \"b10d89f7-d332-4aa4-a06a-7c076f01bc31\") "
Nov 03 19:30:17.469226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.469131    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca" pod "b10d89f7-d332-4aa4-a06a-7c076f01bc31" (UID: "b10d89f7-d332-4aa4-a06a-7c076f01bc31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:17.469226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.469207    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10d89f7-d332-4aa4-a06a-7c076f01bc31-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.470805 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.470765    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz" pod "b10d89f7-d332-4aa4-a06a-7c076f01bc31" (UID: "b10d89f7-d332-4aa4-a06a-7c076f01bc31"). InnerVolumeSpecName "kube-api-access-gkvwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:17.550402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.550374    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:17.570120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.570090    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\" (UniqueName: \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\") pod \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") "
Nov 03 19:30:17.570277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.570146    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\") pod \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\" (UID: \"fef8bffc-dbb4-44cb-be80-36f2d461dbf5\") "
Nov 03 19:30:17.570527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.570503    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca" pod "fef8bffc-dbb4-44cb-be80-36f2d461dbf5" (UID: "fef8bffc-dbb4-44cb-be80-36f2d461dbf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:17.570657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.570589    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.570657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.570603    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkvwz\" (UniqueName: \"kubernetes.io/projected/b10d89f7-d332-4aa4-a06a-7c076f01bc31-kube-api-access-gkvwz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.572305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.572281    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd" pod "fef8bffc-dbb4-44cb-be80-36f2d461dbf5" (UID: "fef8bffc-dbb4-44cb-be80-36f2d461dbf5"). InnerVolumeSpecName "kube-api-access-2hkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:17.607539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.607510    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:17.671822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.671757    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\") pod \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") "
Nov 03 19:30:17.672014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.671885    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\" (UniqueName: \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\") pod \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\" (UID: \"a23977d6-9107-4d07-9b2b-34d9c9d49ff8\") "
Nov 03 19:30:17.672184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.672154    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca" pod "a23977d6-9107-4d07-9b2b-34d9c9d49ff8" (UID: "a23977d6-9107-4d07-9b2b-34d9c9d49ff8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:17.672631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.672607    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.672740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.672634    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hkvd\" (UniqueName: \"kubernetes.io/projected/fef8bffc-dbb4-44cb-be80-36f2d461dbf5-kube-api-access-2hkvd\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.674103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.674073    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z" pod "a23977d6-9107-4d07-9b2b-34d9c9d49ff8" (UID: "a23977d6-9107-4d07-9b2b-34d9c9d49ff8"). InnerVolumeSpecName "kube-api-access-gxn2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:17.774334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.774299    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxn2z\" (UniqueName: \"kubernetes.io/projected/a23977d6-9107-4d07-9b2b-34d9c9d49ff8-kube-api-access-gxn2z\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:17.934671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.934642    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" path="/var/lib/kubelet/pods/317bcf7d-984b-4eb5-b584-5a6c9c7dbe10/volumes"
Nov 03 19:30:17.935189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.935173    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5953eb78-79bc-4e43-800c-77752ef81057" path="/var/lib/kubelet/pods/5953eb78-79bc-4e43-800c-77752ef81057/volumes"
Nov 03 19:30:17.936462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:17.936426    4944 scope.go:122] "RemoveContainer" containerID="3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"
Nov 03 19:30:18.094884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.094780    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"
Nov 03 19:30:18.094884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.094816    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg" event={"ID":"b10d89f7-d332-4aa4-a06a-7c076f01bc31","Type":"ContainerDied","Data":"f652f24969f6eba6df793e8fcd98851eef7ba9a4dec075a320cad422b3186783"}
Nov 03 19:30:18.094884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.094882    4944 scope.go:122] "RemoveContainer" containerID="72826b1321eb2ad0c21617d426af0970de0a42423f24d04cfd4c2b9220b44d3b"
Nov 03 19:30:18.096520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.096487    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"
Nov 03 19:30:18.096520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.096508    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2" event={"ID":"a23977d6-9107-4d07-9b2b-34d9c9d49ff8","Type":"ContainerDied","Data":"d34a858a63203c38937f0cb7d2c1dca3f0543a5bd017507801e2fb83c9b2f8cd"}
Nov 03 19:30:18.098847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.098825    4944 generic.go:356] "Generic (PLEG): container finished" podID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerID="9d25cfb38a82f7bd5cffc5b5519fe8ab8c9f8150498dc31b5fafcbae93a1d636" exitCode=60
Nov 03 19:30:18.098988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.098889    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerDied","Data":"9d25cfb38a82f7bd5cffc5b5519fe8ab8c9f8150498dc31b5fafcbae93a1d636"}
Nov 03 19:30:18.099328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.099273    4944 scope.go:122] "RemoveContainer" containerID="9d25cfb38a82f7bd5cffc5b5519fe8ab8c9f8150498dc31b5fafcbae93a1d636"
Nov 03 19:30:18.099538 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:18.099497    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7_odh-ai-gateway-infra(1a8fe5ec-7703-4034-b56a-11a061c658e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2"
Nov 03 19:30:18.102247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.102228    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg" event={"ID":"fef8bffc-dbb4-44cb-be80-36f2d461dbf5","Type":"ContainerDied","Data":"dc0b052bfa1a903b8929dcff1cf1a61479e4bdeea9ccc59779651f708c741d87"}
Nov 03 19:30:18.102356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.102257    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"
Nov 03 19:30:18.106408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.106390    4944 scope.go:122] "RemoveContainer" containerID="2143c450db60050e7f9888536b01b568fe6525e6fcb5ab98e24fa6c617ae3d03"
Nov 03 19:30:18.125271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.125248    4944 scope.go:122] "RemoveContainer" containerID="a3471f501318b22260c555a794fdb133b46b625d6e9ce947c41568e207c9025e"
Nov 03 19:30:18.138148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.138125    4944 scope.go:122] "RemoveContainer" containerID="292627a228d07c0e58c92a09cbff374820c46c42af98f650f26c6d191d7a4ff6"
Nov 03 19:30:18.310418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.310382    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:18.366929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.366900    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:18.382885 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.382853    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-826a68-29895570-42hsg"]
Nov 03 19:30:18.452691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.452667    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:18.478719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.478686    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:18.482638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.482612    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-010ed3-29895570-2czsg"]
Nov 03 19:30:18.547379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.547345    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:18.584363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.584323    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\") pod \"704e7b63-53b7-4f29-9bde-9840e56dfc06\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") "
Nov 03 19:30:18.584530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.584401    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\") pod \"704e7b63-53b7-4f29-9bde-9840e56dfc06\" (UID: \"704e7b63-53b7-4f29-9bde-9840e56dfc06\") "
Nov 03 19:30:18.584894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.584868    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca" pod "704e7b63-53b7-4f29-9bde-9840e56dfc06" (UID: "704e7b63-53b7-4f29-9bde-9840e56dfc06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:18.587512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.587487    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9" pod "704e7b63-53b7-4f29-9bde-9840e56dfc06" (UID: "704e7b63-53b7-4f29-9bde-9840e56dfc06"). InnerVolumeSpecName "kube-api-access-rwdv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:18.588135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.588115    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:18.597401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.597370    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895570-8lnt2"]
Nov 03 19:30:18.685883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.685853    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwdv9\" (UniqueName: \"kubernetes.io/projected/704e7b63-53b7-4f29-9bde-9840e56dfc06-kube-api-access-rwdv9\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:18.686001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.685887    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/704e7b63-53b7-4f29-9bde-9840e56dfc06-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:18.701403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.701379    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:18.786530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.786500    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\" (UniqueName: \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\") pod \"0fd9cd22-d452-44bb-93de-16e363d24f99\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") "
Nov 03 19:30:18.786658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.786602    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\") pod \"0fd9cd22-d452-44bb-93de-16e363d24f99\" (UID: \"0fd9cd22-d452-44bb-93de-16e363d24f99\") "
Nov 03 19:30:18.787008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.786986    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca" pod "0fd9cd22-d452-44bb-93de-16e363d24f99" (UID: "0fd9cd22-d452-44bb-93de-16e363d24f99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:18.787306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.787288    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd9cd22-d452-44bb-93de-16e363d24f99-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:18.788359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.788340    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv" pod "0fd9cd22-d452-44bb-93de-16e363d24f99" (UID: "0fd9cd22-d452-44bb-93de-16e363d24f99"). InnerVolumeSpecName "kube-api-access-pt6bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:18.888263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:18.888222    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6bv\" (UniqueName: \"kubernetes.io/projected/0fd9cd22-d452-44bb-93de-16e363d24f99-kube-api-access-pt6bv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:19.115589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.115552    4944 generic.go:356] "Generic (PLEG): container finished" podID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerID="730ffd1b22fa145e498a59ba62d6898b62b0fa39b6df236bc2610b6adcb3d420" exitCode=60
Nov 03 19:30:19.116048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.115621    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerDied","Data":"730ffd1b22fa145e498a59ba62d6898b62b0fa39b6df236bc2610b6adcb3d420"}
Nov 03 19:30:19.116048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.115678    4944 scope.go:122] "RemoveContainer" containerID="3dc9a65dcbf8b898ac15a055f12a395819f92c366050f2ed0886bd22aa1acae1"
Nov 03 19:30:19.116164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.116082    4944 scope.go:122] "RemoveContainer" containerID="730ffd1b22fa145e498a59ba62d6898b62b0fa39b6df236bc2610b6adcb3d420"
Nov 03 19:30:19.116403 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:19.116380    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv_odh-ai-gateway-infra(3955a2c1-b2f8-4365-b32e-085ee6a771a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8"
Nov 03 19:30:19.117164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.117141    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"
Nov 03 19:30:19.117245 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.117188    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b" event={"ID":"704e7b63-53b7-4f29-9bde-9840e56dfc06","Type":"ContainerDied","Data":"7f6cbbca4eed2636d6d55d656dda61897947d0e4c5f1f896a14b177b34866ebd"}
Nov 03 19:30:19.119339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.119320    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp" event={"ID":"0fd9cd22-d452-44bb-93de-16e363d24f99","Type":"ContainerDied","Data":"e5b3b1a7343162843ce50bef1a3ae4d8a7f127398ceba9cdad4825e936a76d28"}
Nov 03 19:30:19.119429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.119349    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"
Nov 03 19:30:19.129404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.129307    4944 scope.go:122] "RemoveContainer" containerID="ae93c58bf223d4daabbc3035d39d38f5ee7a724602f87495afe1cfc38883c2f3"
Nov 03 19:30:19.138105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.138090    4944 scope.go:122] "RemoveContainer" containerID="19b442026cf09836a0da878c20486947b2f864292fde331949e02d4fce7266ff"
Nov 03 19:30:19.267074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.267042    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:19.333131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.333099    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:19.344949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.344918    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c3e530-29895570-5qcfp"]
Nov 03 19:30:19.414642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.414612    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:19.443930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.443899    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:19.447694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.447671    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-332dac-29895570-9vb6b"]
Nov 03 19:30:19.495309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.495284    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\" (UniqueName: \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\") pod \"1a8fe5ec-7703-4034-b56a-11a061c658e2\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") "
Nov 03 19:30:19.495455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.495401    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\") pod \"1a8fe5ec-7703-4034-b56a-11a061c658e2\" (UID: \"1a8fe5ec-7703-4034-b56a-11a061c658e2\") "
Nov 03 19:30:19.495727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.495706    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca" pod "1a8fe5ec-7703-4034-b56a-11a061c658e2" (UID: "1a8fe5ec-7703-4034-b56a-11a061c658e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:19.497391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.497369    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp" pod "1a8fe5ec-7703-4034-b56a-11a061c658e2" (UID: "1a8fe5ec-7703-4034-b56a-11a061c658e2"). InnerVolumeSpecName "kube-api-access-nsvlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:19.596416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.596343    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsvlp\" (UniqueName: \"kubernetes.io/projected/1a8fe5ec-7703-4034-b56a-11a061c658e2-kube-api-access-nsvlp\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:19.596416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.596373    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a8fe5ec-7703-4034-b56a-11a061c658e2-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:19.935261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.935178    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" path="/var/lib/kubelet/pods/0fd9cd22-d452-44bb-93de-16e363d24f99/volumes"
Nov 03 19:30:19.935643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.935629    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" path="/var/lib/kubelet/pods/704e7b63-53b7-4f29-9bde-9840e56dfc06/volumes"
Nov 03 19:30:19.936047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.936033    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" path="/var/lib/kubelet/pods/a23977d6-9107-4d07-9b2b-34d9c9d49ff8/volumes"
Nov 03 19:30:19.936710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.936689    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" path="/var/lib/kubelet/pods/b10d89f7-d332-4aa4-a06a-7c076f01bc31/volumes"
Nov 03 19:30:19.937117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:19.937103    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" path="/var/lib/kubelet/pods/fef8bffc-dbb4-44cb-be80-36f2d461dbf5/volumes"
Nov 03 19:30:20.128960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.128928    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7" event={"ID":"1a8fe5ec-7703-4034-b56a-11a061c658e2","Type":"ContainerDied","Data":"fb95ae50dc10b157246b766da59f2b98c4cf0afa15c1423b8963610a730bce7a"}
Nov 03 19:30:20.129388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.128988    4944 scope.go:122] "RemoveContainer" containerID="9d25cfb38a82f7bd5cffc5b5519fe8ab8c9f8150498dc31b5fafcbae93a1d636"
Nov 03 19:30:20.129388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.128936    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"
Nov 03 19:30:20.234772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.234688    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:20.285529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.285500    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:20.301242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.301209    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-9dcd6c-29895570-h7dg7"]
Nov 03 19:30:20.404929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.404902    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:20.506952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.506926    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\") pod \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") "
Nov 03 19:30:20.507096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.507005    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\" (UniqueName: \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\") pod \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\" (UID: \"3955a2c1-b2f8-4365-b32e-085ee6a771a8\") "
Nov 03 19:30:20.507258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.507238    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca" pod "3955a2c1-b2f8-4365-b32e-085ee6a771a8" (UID: "3955a2c1-b2f8-4365-b32e-085ee6a771a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:20.509006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.508980    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq" pod "3955a2c1-b2f8-4365-b32e-085ee6a771a8" (UID: "3955a2c1-b2f8-4365-b32e-085ee6a771a8"). InnerVolumeSpecName "kube-api-access-577dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:20.608765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.608732    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3955a2c1-b2f8-4365-b32e-085ee6a771a8-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:20.608765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:20.608761    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-577dq\" (UniqueName: \"kubernetes.io/projected/3955a2c1-b2f8-4365-b32e-085ee6a771a8-kube-api-access-577dq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:21.140809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.140754    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"
Nov 03 19:30:21.141251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.140754    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv" event={"ID":"3955a2c1-b2f8-4365-b32e-085ee6a771a8","Type":"ContainerDied","Data":"d34a9141a92ead7a7b61448edebc4c8b09463f2bca322af04d60022706cff289"}
Nov 03 19:30:21.141251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.140871    4944 scope.go:122] "RemoveContainer" containerID="730ffd1b22fa145e498a59ba62d6898b62b0fa39b6df236bc2610b6adcb3d420"
Nov 03 19:30:21.296004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.295973    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:21.342330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.342299    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-619731-29895570-nnpjv"]
Nov 03 19:30:21.786125 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:21.786090    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:21.846872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.846825    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:21.940865 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:21.940819    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:21.944085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.944053    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" path="/var/lib/kubelet/pods/1a8fe5ec-7703-4034-b56a-11a061c658e2/volumes"
Nov 03 19:30:21.944545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:21.944525    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" path="/var/lib/kubelet/pods/3955a2c1-b2f8-4365-b32e-085ee6a771a8/volumes"
Nov 03 19:30:23.618197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:23.618127    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerName="main" containerID="cri-o://acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975" gracePeriod=45
Nov 03 19:30:23.633013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:23.632985    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:23.713115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:23.713076    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.219:8000/ready\": dial tcp 10.134.0.219:8000: connect: connection refused"
Nov 03 19:30:23.883587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:23.883523    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:23.887122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:23.887099    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:30:24.048776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.048735    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.048971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.048804    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.048971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.048860    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.048971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.048949    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\" (UniqueName: \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.049175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.048979    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.049175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049003    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\") pod \"d08518d4-62bf-4e80-8a01-3c052ff3030d\" (UID: \"d08518d4-62bf-4e80-8a01-3c052ff3030d\") "
Nov 03 19:30:24.049175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049036    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:30:24.049175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049081    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:30:24.049382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049345    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:30:24.049653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049635    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.049754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049656    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.049754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.049665    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.051295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.051274    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "kube-api-access-97rsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:24.051468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.051453    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:30:24.051513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.051473    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs" pod "d08518d4-62bf-4e80-8a01-3c052ff3030d" (UID: "d08518d4-62bf-4e80-8a01-3c052ff3030d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:30:24.145607 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.145524    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:24.151111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.151089    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d08518d4-62bf-4e80-8a01-3c052ff3030d-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.151182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.151115    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rsv\" (UniqueName: \"kubernetes.io/projected/d08518d4-62bf-4e80-8a01-3c052ff3030d-kube-api-access-97rsv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.151182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.151125    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d08518d4-62bf-4e80-8a01-3c052ff3030d-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:24.164398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.164369    4944 generic.go:356] "Generic (PLEG): container finished" podID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerID="acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975" exitCode=0
Nov 03 19:30:24.164507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.164410    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"
Nov 03 19:30:24.164507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.164419    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" event={"ID":"d08518d4-62bf-4e80-8a01-3c052ff3030d","Type":"ContainerDied","Data":"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"}
Nov 03 19:30:24.164507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.164454    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd" event={"ID":"d08518d4-62bf-4e80-8a01-3c052ff3030d","Type":"ContainerDied","Data":"dab77176522d93fc05a0063d7a2f58444a362e7fd7ef33bf8cc4ad6693f808e8"}
Nov 03 19:30:24.164507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.164471    4944 scope.go:122] "RemoveContainer" containerID="acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"
Nov 03 19:30:24.176891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.176870    4944 scope.go:122] "RemoveContainer" containerID="acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"
Nov 03 19:30:24.177182 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:24.177155    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975\": container with ID starting with acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975 not found: ID does not exist" containerID="acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"
Nov 03 19:30:24.177242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.177192    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975"} err="failed to get container status \"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975\": rpc error: code = NotFound desc = could not find container \"acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975\": container with ID starting with acdc283f8e24a6fc482241da401002fff7c9801815d8084a561631bda7283975 not found: ID does not exist"
Nov 03 19:30:24.318332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.318296    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:30:24.323897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.323867    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6413ba/test-model-6413ba-kserve-84fb4c6df-6chzd"]
Nov 03 19:30:24.423905 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.423823    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"]
Nov 03 19:30:24.424408 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424374    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424473    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424495    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424502    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424522    4944 cpu_manager.go:414] "containerMap: removing container" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424530    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424539    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424545    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424556    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424562    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424575    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424581    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424594    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424601    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.424609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.424615    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425007    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425045    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425054    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425066    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2a19da2f-0cda-4463-9856-e8be2dd69c44" containerName="revoke-keys"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425074    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a19da2f-0cda-4463-9856-e8be2dd69c44" containerName="revoke-keys"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425084    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425091    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425136    4944 cpu_manager.go:414] "containerMap: removing container" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425145    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425155    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425164    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425174    4944 cpu_manager.go:414] "containerMap: removing container" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425182    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425203    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425210    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425222    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerName="main"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425229    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerName="main"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425243    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425250    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425263    4944 cpu_manager.go:414] "containerMap: removing container" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425271    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425298    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425305    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425396    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.425475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425404    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425565    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425600    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425613    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425624    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425633    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a19da2f-0cda-4463-9856-e8be2dd69c44" containerName="revoke-keys"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425641    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425650    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425658    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23977d6-9107-4d07-9b2b-34d9c9d49ff8" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425669    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425681    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425690    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" containerName="main"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425698    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425707    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425717    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425727    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425738    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425746    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425755    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425766    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:24.426698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.425774    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:24.431028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.431006    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.480769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.480736    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:30:24.480938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.480851    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:30:24.482920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.482895    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"]
Nov 03 19:30:24.498135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.498108    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:30:24.498437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.498417    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:30:24.556947 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.556916    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.557167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.557148    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.557241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.557230    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gfb\" (UniqueName: \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.658067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.658038    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.658478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.658087    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gfb\" (UniqueName: \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.658478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.658164    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.658835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.658782    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.660685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.660666    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.715319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.715248    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gfb\" (UniqueName: \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\") pod \"maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:24.744661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:24.744631    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:25.117512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:25.117475    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"]
Nov 03 19:30:25.123782 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:25.123747    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9623531d_800e_4ba3_a941_a1bc93fddd06.slice/crio-c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083 WatchSource:0}: Error finding container c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083: Status 404 returned error can't find the container with id c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083
Nov 03 19:30:25.175003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:25.174967    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv" event={"ID":"9623531d-800e-4ba3-a941-a1bc93fddd06","Type":"ContainerStarted","Data":"c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083"}
Nov 03 19:30:25.935664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:25.935623    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08518d4-62bf-4e80-8a01-3c052ff3030d" path="/var/lib/kubelet/pods/d08518d4-62bf-4e80-8a01-3c052ff3030d/volumes"
Nov 03 19:30:26.188597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:26.188502    4944 generic.go:356] "Generic (PLEG): container finished" podID="9623531d-800e-4ba3-a941-a1bc93fddd06" containerID="6d21ff123e0fe8b6ce581d94f37661ee5f3dec53e09c6339dd8796de3292955e" exitCode=0
Nov 03 19:30:26.188597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:26.188557    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv" event={"ID":"9623531d-800e-4ba3-a941-a1bc93fddd06","Type":"ContainerDied","Data":"6d21ff123e0fe8b6ce581d94f37661ee5f3dec53e09c6339dd8796de3292955e"}
Nov 03 19:30:26.647539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:26.647494    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:27.333725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.333697    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:27.406406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.406372    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\") pod \"9623531d-800e-4ba3-a941-a1bc93fddd06\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") "
Nov 03 19:30:27.406566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.406422    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\") pod \"9623531d-800e-4ba3-a941-a1bc93fddd06\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") "
Nov 03 19:30:27.406566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.406451    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\" (UniqueName: \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\") pod \"9623531d-800e-4ba3-a941-a1bc93fddd06\" (UID: \"9623531d-800e-4ba3-a941-a1bc93fddd06\") "
Nov 03 19:30:27.406815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.406768    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca" pod "9623531d-800e-4ba3-a941-a1bc93fddd06" (UID: "9623531d-800e-4ba3-a941-a1bc93fddd06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:27.406926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.406906    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:27.408465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.408442    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca" pod "9623531d-800e-4ba3-a941-a1bc93fddd06" (UID: "9623531d-800e-4ba3-a941-a1bc93fddd06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:30:27.408552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.408511    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb" pod "9623531d-800e-4ba3-a941-a1bc93fddd06" (UID: "9623531d-800e-4ba3-a941-a1bc93fddd06"). InnerVolumeSpecName "kube-api-access-k9gfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:27.508506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.508473    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9623531d-800e-4ba3-a941-a1bc93fddd06-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:27.508506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:27.508502    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9gfb\" (UniqueName: \"kubernetes.io/projected/9623531d-800e-4ba3-a941-a1bc93fddd06-kube-api-access-k9gfb\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:28.204315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:28.204229    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"
Nov 03 19:30:28.204315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:28.204267    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv" event={"ID":"9623531d-800e-4ba3-a941-a1bc93fddd06","Type":"ContainerDied","Data":"c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083"}
Nov 03 19:30:28.204315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:28.204315    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64361fd5b888bf98b7687cff61480a1b0d0f8da9d4b4dda9d6d29e09a749083"
Nov 03 19:30:31.348078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:31.348010    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:31.854175 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:31.854129    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:32.581081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:32.581042    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:33.051467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:33.051430    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"]
Nov 03 19:30:33.097926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:33.097892    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ff2d7d2-dqb59"]
Nov 03 19:30:33.934298 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:33.934260    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6676cea4-3aa2-4163-996d-27a1b2f56b8d" path="/var/lib/kubelet/pods/6676cea4-3aa2-4163-996d-27a1b2f56b8d/volumes"
Nov 03 19:30:34.869842 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:34.869805    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:34.870017 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:34.869957    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:35.225319 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:35.225223    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:30:35.907571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:35.907534    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:36.975198 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:36.975167    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:37.159924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:37.159887    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:37.409125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:37.409092    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:40.466231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:40.466195    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:41.867562 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:41.867473    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:44.987896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:44.987862    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:46.086081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:46.086038    4944 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod317bcf7d-984b-4eb5-b584-5a6c9c7dbe10"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod317bcf7d-984b-4eb5-b584-5a6c9c7dbe10] : Timed out while waiting for systemd to remove kubepods-burstable-pod317bcf7d_984b_4eb5_b584_5a6c9c7dbe10.slice"
Nov 03 19:30:46.086081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:46.086075    4944 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="317bcf7d-984b-4eb5-b584-5a6c9c7dbe10" err="failed to delete cgroup paths for [kubepods burstable pod317bcf7d-984b-4eb5-b584-5a6c9c7dbe10] : unable to destroy cgroup paths for cgroup [kubepods burstable pod317bcf7d-984b-4eb5-b584-5a6c9c7dbe10] : Timed out while waiting for systemd to remove kubepods-burstable-pod317bcf7d_984b_4eb5_b584_5a6c9c7dbe10.slice"
Nov 03 19:30:49.670720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:49.670684    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:50.823143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823110    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"]
Nov 03 19:30:50.823697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823682    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9623531d-800e-4ba3-a941-a1bc93fddd06" containerName="revoke-keys"
Nov 03 19:30:50.823747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823699    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9623531d-800e-4ba3-a941-a1bc93fddd06" containerName="revoke-keys"
Nov 03 19:30:50.823747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823718    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:50.823747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823724    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:50.823747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823731    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:50.823747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823736    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823752    4944 cpu_manager.go:414] "containerMap: removing container" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823757    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823769    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823774    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8fe5ec-7703-4034-b56a-11a061c658e2" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823783    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823803    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823911    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9623531d-800e-4ba3-a941-a1bc93fddd06" containerName="revoke-keys"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823918    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="704e7b63-53b7-4f29-9bde-9840e56dfc06" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823925    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd9cd22-d452-44bb-93de-16e363d24f99" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823931    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3955a2c1-b2f8-4365-b32e-085ee6a771a8" containerName="cleanup"
Nov 03 19:30:50.823935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823938    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef8bffc-dbb4-44cb-be80-36f2d461dbf5" containerName="cleanup"
Nov 03 19:30:50.824288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.823948    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10d89f7-d332-4aa4-a06a-7c076f01bc31" containerName="cleanup"
Nov 03 19:30:50.830073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.830047    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:50.881069 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.881039    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"]
Nov 03 19:30:50.923047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.923012    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:30:50.923047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.923014    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:30:50.923246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.923059    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:30:50.924696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.924674    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:30:50.987590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.987556    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:50.987727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.987594    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hvgl\" (UniqueName: \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:50.987727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:50.987709    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.088774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.088683    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.088955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.088919    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.089022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.088960    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hvgl\" (UniqueName: \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.089443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.089417    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.091230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.091210    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.146969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.146938    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hvgl\" (UniqueName: \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\") pod \"maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.428389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.428303    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:30:51.443518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.443488    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:51.581453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.581375    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:51.817444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:51.817404    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"]
Nov 03 19:30:51.828491 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:30:51.828456    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34872d7f_2971_4482_ad92_ae24b8bdbe09.slice/crio-a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f WatchSource:0}: Error finding container a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f: Status 404 returned error can't find the container with id a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f
Nov 03 19:30:51.935757 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:51.935713    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:51.935950 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:30:51.935781    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:30:52.378742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:52.378700    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc" event={"ID":"34872d7f-2971-4482-ad92-ae24b8bdbe09","Type":"ContainerStarted","Data":"233f1dd3d40e655b59aa5849508b1b6d7e16d07e4d0cc4fb5638d97d3f8f949e"}
Nov 03 19:30:52.378938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:52.378797    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc" event={"ID":"34872d7f-2971-4482-ad92-ae24b8bdbe09","Type":"ContainerStarted","Data":"a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f"}
Nov 03 19:30:52.487039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:52.486939    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc" podStartSLOduration=2.486924058 podStartE2EDuration="2.486924058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:30:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:30:52.486321103 +0000 UTC m=+2750.763674683" watchObservedRunningTime="2026-11-03 19:30:52.486924058 +0000 UTC m=+2750.764277635"
Nov 03 19:30:53.386435 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:53.386402    4944 generic.go:356] "Generic (PLEG): container finished" podID="34872d7f-2971-4482-ad92-ae24b8bdbe09" containerID="233f1dd3d40e655b59aa5849508b1b6d7e16d07e4d0cc4fb5638d97d3f8f949e" exitCode=0
Nov 03 19:30:53.386898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:53.386485    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc" event={"ID":"34872d7f-2971-4482-ad92-ae24b8bdbe09","Type":"ContainerDied","Data":"233f1dd3d40e655b59aa5849508b1b6d7e16d07e4d0cc4fb5638d97d3f8f949e"}
Nov 03 19:30:53.819446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:53.819416    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:54.543262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.543233    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:54.544765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.544742    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:54.638299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.638270    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\") pod \"34872d7f-2971-4482-ad92-ae24b8bdbe09\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") "
Nov 03 19:30:54.638450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.638374    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\" (UniqueName: \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\") pod \"34872d7f-2971-4482-ad92-ae24b8bdbe09\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") "
Nov 03 19:30:54.638450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.638430    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\") pod \"34872d7f-2971-4482-ad92-ae24b8bdbe09\" (UID: \"34872d7f-2971-4482-ad92-ae24b8bdbe09\") "
Nov 03 19:30:54.638686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.638664    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca" pod "34872d7f-2971-4482-ad92-ae24b8bdbe09" (UID: "34872d7f-2971-4482-ad92-ae24b8bdbe09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:30:54.638903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.638887    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:54.640556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.640532    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca" pod "34872d7f-2971-4482-ad92-ae24b8bdbe09" (UID: "34872d7f-2971-4482-ad92-ae24b8bdbe09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:30:54.640652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.640599    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl" pod "34872d7f-2971-4482-ad92-ae24b8bdbe09" (UID: "34872d7f-2971-4482-ad92-ae24b8bdbe09"). InnerVolumeSpecName "kube-api-access-5hvgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:30:54.740170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.740136    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hvgl\" (UniqueName: \"kubernetes.io/projected/34872d7f-2971-4482-ad92-ae24b8bdbe09-kube-api-access-5hvgl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:54.740170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:54.740167    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34872d7f-2971-4482-ad92-ae24b8bdbe09-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:30:55.403144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:55.403111    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"
Nov 03 19:30:55.403364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:55.403142    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc" event={"ID":"34872d7f-2971-4482-ad92-ae24b8bdbe09","Type":"ContainerDied","Data":"a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f"}
Nov 03 19:30:55.403364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:55.403192    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5ba04e9b07d449ec4cc5b800427598e6acc756f3f4b0921f03d73383df3d90f"
Nov 03 19:30:59.291535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:59.291480    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:59.589668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:59.589583    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:30:59.836540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:30:59.836509    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:01.728064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:01.728023    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:01.994187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:01.994155    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:02.010367 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:02.010329    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8fe5ec_7703_4034_b56a_11a061c658e2.slice/crio-1c0179b17e4b41453ff7fe858d82b8adb86dea699afd06551ba83bd99bd9683a.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 19:31:04.174342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:04.174281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:04.949972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:04.949936    4944 scope.go:122] "RemoveContainer" containerID="31ffc99db66f78d4ecfdd40dab57693c9f66e116930a05c0292080124ecd2890"
Nov 03 19:31:04.959771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:04.959746    4944 scope.go:122] "RemoveContainer" containerID="c2ac2fc62c92235b990a56e459f1dda66a9cffa2b3d8a313b430ce3849a4ca52"
Nov 03 19:31:04.969453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:04.969430    4944 scope.go:122] "RemoveContainer" containerID="62ca77cb7fd47dcbf8ad6ff9639e11f2edfe7b31f4e5f156b724f7f7223dc13e"
Nov 03 19:31:04.978427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:04.978404    4944 scope.go:122] "RemoveContainer" containerID="cd81ee739f98cedd0a9d2ecb05909810e836565af873ed258ec3ac584a554c04"
Nov 03 19:31:05.224329 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:05.224248    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:31:06.451513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.451470    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" podUID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" containerName="main" containerID="cri-o://a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2" gracePeriod=45
Nov 03 19:31:06.722446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.722418    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:31:06.809634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809604    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.809836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809669    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.809836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809742    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.809836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809817    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\" (UniqueName: \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.810035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809879    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\" (UniqueName: \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.810035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809917    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\") pod \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\" (UID: \"680f8fd7-dd37-47d7-868f-2878bfd4ff90\") "
Nov 03 19:31:06.810035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809913    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:06.810186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809929    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:06.810186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.809990    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:06.810467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.810446    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:06.810555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.810472    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:06.810555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.810486    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:06.812110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.812084    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:06.812226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.812135    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:06.812226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.812200    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv" pod "680f8fd7-dd37-47d7-868f-2878bfd4ff90" (UID: "680f8fd7-dd37-47d7-868f-2878bfd4ff90"). InnerVolumeSpecName "kube-api-access-w56xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:06.911648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.911598    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680f8fd7-dd37-47d7-868f-2878bfd4ff90-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:06.911648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.911639    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680f8fd7-dd37-47d7-868f-2878bfd4ff90-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:06.911648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:06.911654    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w56xv\" (UniqueName: \"kubernetes.io/projected/680f8fd7-dd37-47d7-868f-2878bfd4ff90-kube-api-access-w56xv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:07.481497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.481459    4944 generic.go:356] "Generic (PLEG): container finished" podID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" containerID="a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2" exitCode=0
Nov 03 19:31:07.481955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.481507    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"
Nov 03 19:31:07.481955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.481544    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" event={"ID":"680f8fd7-dd37-47d7-868f-2878bfd4ff90","Type":"ContainerDied","Data":"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"}
Nov 03 19:31:07.481955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.481591    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd" event={"ID":"680f8fd7-dd37-47d7-868f-2878bfd4ff90","Type":"ContainerDied","Data":"5d41ca86c7f8c761d1b832a10481975846f80434fc7c16ad07e3053970d47920"}
Nov 03 19:31:07.481955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.481608    4944 scope.go:122] "RemoveContainer" containerID="a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"
Nov 03 19:31:07.491729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.491710    4944 scope.go:122] "RemoveContainer" containerID="a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"
Nov 03 19:31:07.492017 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:07.491997    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2\": container with ID starting with a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2 not found: ID does not exist" containerID="a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"
Nov 03 19:31:07.492072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.492024    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2"} err="failed to get container status \"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2\": rpc error: code = NotFound desc = could not find container \"a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2\": container with ID starting with a3dac2fcacaa7da7cff4ebfe6bfe925bd2f08eb5cfe569dc5e42bfa59c6401f2 not found: ID does not exist"
Nov 03 19:31:07.635342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.635309    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:31:07.681453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.681422    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c3e530/test-model-c3e530-kserve-58c4cc6c57-zffbd"]
Nov 03 19:31:07.934185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:07.934152    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" path="/var/lib/kubelet/pods/680f8fd7-dd37-47d7-868f-2878bfd4ff90/volumes"
Nov 03 19:31:08.711122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:08.711084    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:13.232548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:13.232517    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:17.150398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:17.150355    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:18.697688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:18.697629    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:18.967056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:18.966967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.070028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:19.069991    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"]
Nov 03 19:31:19.115717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:19.115682    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6413ba-9hrbc"]
Nov 03 19:31:19.213738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:19.213704    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.572362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:19.572331    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:19.934336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:19.934256    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34872d7f-2971-4482-ad92-ae24b8bdbe09" path="/var/lib/kubelet/pods/34872d7f-2971-4482-ad92-ae24b8bdbe09/volumes"
Nov 03 19:31:23.108150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:23.108113    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:24.661739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:24.661700    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:24.999935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:24.999890    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.098751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.098717    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"]
Nov 03 19:31:25.145279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.145246    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-332dac-9vdcv"]
Nov 03 19:31:25.243990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.243938    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.607519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.607489    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.853968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.853928    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:25.935445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:25.935369    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9623531d-800e-4ba3-a941-a1bc93fddd06" path="/var/lib/kubelet/pods/9623531d-800e-4ba3-a941-a1bc93fddd06/volumes"
Nov 03 19:31:26.138887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.138851    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:26.611807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.611760    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"]
Nov 03 19:31:26.612579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612553    4944 cpu_manager.go:414] "containerMap: removing container" podUID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" containerName="main"
Nov 03 19:31:26.612579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612579    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" containerName="main"
Nov 03 19:31:26.612820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612601    4944 cpu_manager.go:414] "containerMap: removing container" podUID="34872d7f-2971-4482-ad92-ae24b8bdbe09" containerName="revoke-keys"
Nov 03 19:31:26.612820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612610    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34872d7f-2971-4482-ad92-ae24b8bdbe09" containerName="revoke-keys"
Nov 03 19:31:26.612820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612775    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="680f8fd7-dd37-47d7-868f-2878bfd4ff90" containerName="main"
Nov 03 19:31:26.612820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.612805    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="34872d7f-2971-4482-ad92-ae24b8bdbe09" containerName="revoke-keys"
Nov 03 19:31:26.626172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.626143    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.672981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.672950    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"]
Nov 03 19:31:26.713480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.713449    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:31:26.713480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.713459    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:31:26.713693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.713534    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:31:26.730346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.730291    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:31:26.752460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.751430    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.752460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.751486    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzt6p\" (UniqueName: \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.752460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.751590    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.852822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.852755    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.853014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.852914    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.853014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.852942    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzt6p\" (UniqueName: \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.853619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.853599    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.855076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.855055    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.910475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.910405    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzt6p\" (UniqueName: \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\") pod \"maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:26.962127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:26.962100    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:27.337605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:27.337580    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"]
Nov 03 19:31:27.339413 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:31:27.339380    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c740017_0fbd_4a8b_b75a_fca050322793.slice/crio-d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2 WatchSource:0}: Error finding container d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2: Status 404 returned error can't find the container with id d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2
Nov 03 19:31:27.628623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:27.628529    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc" event={"ID":"5c740017-0fbd-4a8b-b75a-fca050322793","Type":"ContainerStarted","Data":"d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2"}
Nov 03 19:31:28.641181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:28.641097    4944 generic.go:356] "Generic (PLEG): container finished" podID="5c740017-0fbd-4a8b-b75a-fca050322793" containerID="1cba25435172b39d6314a8fc4e1fbdef1199f1d29634b2ca31e61b294370b70e" exitCode=0
Nov 03 19:31:28.641577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:28.641178    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc" event={"ID":"5c740017-0fbd-4a8b-b75a-fca050322793","Type":"ContainerDied","Data":"1cba25435172b39d6314a8fc4e1fbdef1199f1d29634b2ca31e61b294370b70e"}
Nov 03 19:31:29.803277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.803251    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:29.874449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.874417    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:29.893641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.893605    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\") pod \"5c740017-0fbd-4a8b-b75a-fca050322793\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") "
Nov 03 19:31:29.893841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.893783    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\") pod \"5c740017-0fbd-4a8b-b75a-fca050322793\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") "
Nov 03 19:31:29.893934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.893859    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\" (UniqueName: \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\") pod \"5c740017-0fbd-4a8b-b75a-fca050322793\" (UID: \"5c740017-0fbd-4a8b-b75a-fca050322793\") "
Nov 03 19:31:29.894134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.894106    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca" pod "5c740017-0fbd-4a8b-b75a-fca050322793" (UID: "5c740017-0fbd-4a8b-b75a-fca050322793"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:29.894405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.894386    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:29.895900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.895880    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca" pod "5c740017-0fbd-4a8b-b75a-fca050322793" (UID: "5c740017-0fbd-4a8b-b75a-fca050322793"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:29.896003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.895908    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p" pod "5c740017-0fbd-4a8b-b75a-fca050322793" (UID: "5c740017-0fbd-4a8b-b75a-fca050322793"). InnerVolumeSpecName "kube-api-access-pzt6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:29.995839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.995781    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzt6p\" (UniqueName: \"kubernetes.io/projected/5c740017-0fbd-4a8b-b75a-fca050322793-kube-api-access-pzt6p\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:29.995839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:29.995835    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c740017-0fbd-4a8b-b75a-fca050322793-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:30.656188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:30.656158    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"
Nov 03 19:31:30.656373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:30.656192    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc" event={"ID":"5c740017-0fbd-4a8b-b75a-fca050322793","Type":"ContainerDied","Data":"d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2"}
Nov 03 19:31:30.656373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:30.656242    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d97110df4c269cc78d2e1c8b3b529b00f3f4fac2ae0e2c867992a783947d33e2"
Nov 03 19:31:31.730331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:31.730293    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:31.979293 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:31.979261    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:32.233274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.233234    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:32.491331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.491302    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:32.647461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.647431    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"]
Nov 03 19:31:32.648097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.648075    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5c740017-0fbd-4a8b-b75a-fca050322793" containerName="revoke-keys"
Nov 03 19:31:32.648097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.648097    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c740017-0fbd-4a8b-b75a-fca050322793" containerName="revoke-keys"
Nov 03 19:31:32.648378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.648362    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c740017-0fbd-4a8b-b75a-fca050322793" containerName="revoke-keys"
Nov 03 19:31:32.652247 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.652223    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.748017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.747883    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:31:32.748017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.747894    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:31:32.748017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.747975    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:31:32.748489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.748076    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:31:32.751032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.751007    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"]
Nov 03 19:31:32.753761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.753734    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqw7d\" (UniqueName: \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.753897 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.753826    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.754080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.754062    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.855639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.855611    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.855784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.855664    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqw7d\" (UniqueName: \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.855784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.855710    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.856394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.856369    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.858116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.858090    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.913035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.913003    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqw7d\" (UniqueName: \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\") pod \"maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:32.966798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:32.966753    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:33.220013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:33.219977    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:33.344565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:33.344482    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"]
Nov 03 19:31:33.350077 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:31:33.350043    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba44825_3049_4613_a789_fe5931454a43.slice/crio-c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c WatchSource:0}: Error finding container c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c: Status 404 returned error can't find the container with id c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c
Nov 03 19:31:33.676856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:33.676815    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j" event={"ID":"aba44825-3049-4613-a789-fe5931454a43","Type":"ContainerStarted","Data":"943ee3792450ade441da6bf8ae83ef2165d8ee8a92281dd78616a94ea07ccb03"}
Nov 03 19:31:33.677091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:33.676883    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j" event={"ID":"aba44825-3049-4613-a789-fe5931454a43","Type":"ContainerStarted","Data":"c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c"}
Nov 03 19:31:33.786263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:33.786207    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j" podStartSLOduration=1.786192417 podStartE2EDuration="1.786192417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:33.784349068 +0000 UTC m=+2792.061702673" watchObservedRunningTime="2026-11-03 19:31:33.786192417 +0000 UTC m=+2792.063545999"
Nov 03 19:31:34.685747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:34.685660    4944 generic.go:356] "Generic (PLEG): container finished" podID="aba44825-3049-4613-a789-fe5931454a43" containerID="943ee3792450ade441da6bf8ae83ef2165d8ee8a92281dd78616a94ea07ccb03" exitCode=0
Nov 03 19:31:34.685747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:34.685741    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j" event={"ID":"aba44825-3049-4613-a789-fe5931454a43","Type":"ContainerDied","Data":"943ee3792450ade441da6bf8ae83ef2165d8ee8a92281dd78616a94ea07ccb03"}
Nov 03 19:31:34.802318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:34.802281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:35.225479 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:35.225435    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:31:35.842687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.842663    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:35.993418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.993394    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\" (UniqueName: \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\") pod \"aba44825-3049-4613-a789-fe5931454a43\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") "
Nov 03 19:31:35.993593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.993483    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\") pod \"aba44825-3049-4613-a789-fe5931454a43\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") "
Nov 03 19:31:35.993593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.993553    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\") pod \"aba44825-3049-4613-a789-fe5931454a43\" (UID: \"aba44825-3049-4613-a789-fe5931454a43\") "
Nov 03 19:31:35.993872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.993852    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca" pod "aba44825-3049-4613-a789-fe5931454a43" (UID: "aba44825-3049-4613-a789-fe5931454a43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:35.994062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.994035    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aba44825-3049-4613-a789-fe5931454a43-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:35.995631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.995601    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca" pod "aba44825-3049-4613-a789-fe5931454a43" (UID: "aba44825-3049-4613-a789-fe5931454a43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:35.995727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:35.995670    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d" pod "aba44825-3049-4613-a789-fe5931454a43" (UID: "aba44825-3049-4613-a789-fe5931454a43"). InnerVolumeSpecName "kube-api-access-qqw7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:36.095290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.095254    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqw7d\" (UniqueName: \"kubernetes.io/projected/aba44825-3049-4613-a789-fe5931454a43-kube-api-access-qqw7d\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:36.095290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.095284    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aba44825-3049-4613-a789-fe5931454a43-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:36.700013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.699979    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"
Nov 03 19:31:36.700013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.699990    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j" event={"ID":"aba44825-3049-4613-a789-fe5931454a43","Type":"ContainerDied","Data":"c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c"}
Nov 03 19:31:36.700256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.700038    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c16f0540f3fbb5c710aacb181111cac3f08ab07453b650c9b27dd259a19b142c"
Nov 03 19:31:36.701596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.701571    4944 generic.go:356] "Generic (PLEG): container finished" podID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerID="28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409" exitCode=0
Nov 03 19:31:36.701719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.701639    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerDied","Data":"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"}
Nov 03 19:31:36.702031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.702017    4944 scope.go:122] "RemoveContainer" containerID="28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"
Nov 03 19:31:36.703697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.703665    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:31:36.703772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.703755    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:31:36.703844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:36.703831    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:31:37.352088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.352053    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:31:37.359191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.359160    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:31:37.363601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.363570    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:31:37.370628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.370605    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 19:31:37.560081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.560047    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:37.734176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:37.734075    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerStarted","Data":"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"}
Nov 03 19:31:38.231421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.231164    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:31:38.603950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.603914    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:31:38.604417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.604164    4944 patch_prober.go:29] interesting pod/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 19:31:38.604417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.604251    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 19:31:38.660956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.660888    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:38.932770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:38.932666    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:39.603470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:39.603429    4944 patch_prober.go:29] interesting pod/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 19:31:39.603708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:39.603496    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 19:31:39.673322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:39.673286    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.533453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.533400    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.604031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.603986    4944 patch_prober.go:29] interesting pod/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 19:31:40.604200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.604051    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 19:31:40.782948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.782915    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:40.960193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.960109    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:31:40.960395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:40.960357    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy" containerID="cri-o://8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503" gracePeriod=30
Nov 03 19:31:41.006282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.006239    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 19:31:41.006490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.006467    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" containerID="cri-o://3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014" gracePeriod=30
Nov 03 19:31:41.033917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.033870    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:31:41.034156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.034130    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy" containerID="cri-o://9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6" gracePeriod=30
Nov 03 19:31:41.037693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.037666    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:31:41.037930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.037906    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" containerID="cri-o://308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90" gracePeriod=30
Nov 03 19:31:41.303552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.303514    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-7jmdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:31:41.303722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.303573    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:31:41.394500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.394457    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-4t7v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 19:31:41.394664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.394519    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 19:31:41.603486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.603400    4944 patch_prober.go:29] interesting pod/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 19:31:41.603486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:41.603454    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 19:31:42.603954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:42.603920    4944 patch_prober.go:29] interesting pod/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 19:31:42.604352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:42.603973    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 19:31:43.232523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.232483    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" podUID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" containerName="payload-processing" containerID="cri-o://3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2" gracePeriod=25
Nov 03 19:31:43.500718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.500683    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:31:43.516470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.516443    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\" (UniqueName: \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\") pod \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") "
Nov 03 19:31:43.516629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.516615    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\") pod \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\" (UID: \"f305f266-bb0c-473b-92a1-ebdb03c2f9d7\") "
Nov 03 19:31:43.517030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.517002    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume" pod "f305f266-bb0c-473b-92a1-ebdb03c2f9d7" (UID: "f305f266-bb0c-473b-92a1-ebdb03c2f9d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:43.518971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.518951    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2" pod "f305f266-bb0c-473b-92a1-ebdb03c2f9d7" (UID: "f305f266-bb0c-473b-92a1-ebdb03c2f9d7"). InnerVolumeSpecName "kube-api-access-2nmx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:43.607900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.607873    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:31:43.608275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.608138    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:31:43.609058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.609034    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:31:43.618782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.618758    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:43.618929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.618817    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmx2\" (UniqueName: \"kubernetes.io/projected/f305f266-bb0c-473b-92a1-ebdb03c2f9d7-kube-api-access-2nmx2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:43.782422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.782393    4944 generic.go:356] "Generic (PLEG): container finished" podID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" containerID="3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2" exitCode=0
Nov 03 19:31:43.782611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.782431    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"
Nov 03 19:31:43.782611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.782470    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" event={"ID":"f305f266-bb0c-473b-92a1-ebdb03c2f9d7","Type":"ContainerDied","Data":"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"}
Nov 03 19:31:43.782611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.782514    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt" event={"ID":"f305f266-bb0c-473b-92a1-ebdb03c2f9d7","Type":"ContainerDied","Data":"3fdcf3d17af7edd29ed12eca5049e709ea5b3b56d3455ec01175cc5fd9f4a0b7"}
Nov 03 19:31:43.782611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.782539    4944 scope.go:122] "RemoveContainer" containerID="3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"
Nov 03 19:31:43.792766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.792750    4944 scope.go:122] "RemoveContainer" containerID="3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"
Nov 03 19:31:43.793047 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:43.793028    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2\": container with ID starting with 3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2 not found: ID does not exist" containerID="3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"
Nov 03 19:31:43.793127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.793058    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2"} err="failed to get container status \"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2\": rpc error: code = NotFound desc = could not find container \"3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2\": container with ID starting with 3232c95db3cec2bee07d0c4aef236bfa098732bd8a588019c173560cba5f73f2 not found: ID does not exist"
Nov 03 19:31:43.950324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.950293    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:31:43.954377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:43.954346    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c3e530-74597b6bf6-xqjdt"]
Nov 03 19:31:44.303573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:44.303536    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 19:31:44.462969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:44.462938    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 19:31:44.759956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:44.759922    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:45.011379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:45.011295    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:45.935543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:45.935508    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" path="/var/lib/kubelet/pods/f305f266-bb0c-473b-92a1-ebdb03c2f9d7/volumes"
Nov 03 19:31:46.256720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.256685    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 19:31:46.298531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.298505    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:31:46.350631 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350599    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.350827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350647    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.350827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350682    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.350827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350726    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.350827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350751    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.350827 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350814    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350866    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350895    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350925    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.350995    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351024    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351095    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351133    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351192    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351241    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351337    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351352    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351418    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\") pod \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\" (UID: \"5edde6c1-e803-46f8-aa0a-689f4a2f7838\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351451    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\") pod \"3910b36b-0adb-4567-b50c-a84b2784ce45\" (UID: \"3910b36b-0adb-4567-b50c-a84b2784ce45\") "
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351565    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.351655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351625    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.352135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351747    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.352135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351757    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.352135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.351938    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:46.352773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352743    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.352773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352773    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.353005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352817    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.353005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352834    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.353005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352847    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.353005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.352858    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.355241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.355048    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.355241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.355064    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.355429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.355376    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:46.355578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.355524    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "kube-api-access-bgmtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.355578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.355544    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.356833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.356771    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.357013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.356995    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.357235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.357211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:31:46.357359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.357237    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.358127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.358097    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.359581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.359558    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg" pod "3910b36b-0adb-4567-b50c-a84b2784ce45" (UID: "3910b36b-0adb-4567-b50c-a84b2784ce45"). InnerVolumeSpecName "kube-api-access-qwcvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.359672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.359635    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo" pod "5edde6c1-e803-46f8-aa0a-689f4a2f7838" (UID: "5edde6c1-e803-46f8-aa0a-689f4a2f7838"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:31:46.376317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.376296    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:31:46.385842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.385824    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:31:46.453594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453554    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.453594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453597    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453642    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453670    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453693    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453721    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453760    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453829    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453861    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.453894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453888    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453930    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453955    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.453969    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454002    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454000    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454039    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454091    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454114    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454149    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\") pod \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\" (UID: \"f8a503d9-371d-4265-951c-2cc6b0abcf5c\") "
Nov 03 19:31:46.454334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454172    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\") pod \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\" (UID: \"a9314ad0-e9cc-4aee-867f-2a70190ebbf0\") "
Nov 03 19:31:46.454895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454379    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.454895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454504    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.454895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454704    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.454895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454732    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454897    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454917    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454932    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454933    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwcvg\" (UniqueName: \"kubernetes.io/projected/3910b36b-0adb-4567-b50c-a84b2784ce45-kube-api-access-qwcvg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454962    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3910b36b-0adb-4567-b50c-a84b2784ce45-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454980    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.454997    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmtr\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-kube-api-access-bgmtr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455013    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455017    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455029    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455062    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455079    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455094    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455109    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455124    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455138    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455153    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455168    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edde6c1-e803-46f8-aa0a-689f4a2f7838-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455182    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.455396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455197    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3910b36b-0adb-4567-b50c-a84b2784ce45-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.456503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455488    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.456503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.455917    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.457419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.457378    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.457635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.457601    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.457635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.457613    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.458042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.457997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "kube-api-access-8jght". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.458132 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.458108    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo" pod "f8a503d9-371d-4265-951c-2cc6b0abcf5c" (UID: "f8a503d9-371d-4265-951c-2cc6b0abcf5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:31:46.458362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.458338    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "kube-api-access-t7qzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:46.458711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.458691    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:46.459423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.459408    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo" pod "a9314ad0-e9cc-4aee-867f-2a70190ebbf0" (UID: "a9314ad0-e9cc-4aee-867f-2a70190ebbf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:31:46.556477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556440    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556470    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556481    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556503    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556513    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a503d9-371d-4265-951c-2cc6b0abcf5c-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556522    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556530    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7qzq\" (UniqueName: \"kubernetes.io/projected/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-kube-api-access-t7qzq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556541    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556549    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9314ad0-e9cc-4aee-867f-2a70190ebbf0-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556557    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jght\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-kube-api-access-8jght\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556565    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.556839 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.556573    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a503d9-371d-4265-951c-2cc6b0abcf5c-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:46.810358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.810273    4944 generic.go:356] "Generic (PLEG): container finished" podID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerID="308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90" exitCode=0
Nov 03 19:31:46.810358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.810318    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"
Nov 03 19:31:46.810573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.810358    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" event={"ID":"f8a503d9-371d-4265-951c-2cc6b0abcf5c","Type":"ContainerDied","Data":"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"}
Nov 03 19:31:46.810573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.810420    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2" event={"ID":"f8a503d9-371d-4265-951c-2cc6b0abcf5c","Type":"ContainerDied","Data":"60eddc3134ad83023aaa2609ebb052d0fe05abc6338c6e2d5f41f19658857bc4"}
Nov 03 19:31:46.810573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.810453    4944 scope.go:122] "RemoveContainer" containerID="308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"
Nov 03 19:31:46.812812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.812739    4944 generic.go:356] "Generic (PLEG): container finished" podID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerID="8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503" exitCode=0
Nov 03 19:31:46.812812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.812808    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" event={"ID":"3910b36b-0adb-4567-b50c-a84b2784ce45","Type":"ContainerDied","Data":"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"}
Nov 03 19:31:46.813050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.812854    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72" event={"ID":"3910b36b-0adb-4567-b50c-a84b2784ce45","Type":"ContainerDied","Data":"46edaed97ab8e501ff67925a1626599790e273bf2b5dff207b0731bdd7aa704a"}
Nov 03 19:31:46.813050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.812779    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"
Nov 03 19:31:46.814623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.814575    4944 generic.go:356] "Generic (PLEG): container finished" podID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerID="3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014" exitCode=0
Nov 03 19:31:46.814807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.814633    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"
Nov 03 19:31:46.814881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.814856    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" event={"ID":"5edde6c1-e803-46f8-aa0a-689f4a2f7838","Type":"ContainerDied","Data":"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"}
Nov 03 19:31:46.815015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.814917    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj" event={"ID":"5edde6c1-e803-46f8-aa0a-689f4a2f7838","Type":"ContainerDied","Data":"5509ca00d76a50fd7d86084167a03b1cecce41907a3f219aa2de6616fdc34646"}
Nov 03 19:31:46.817037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.816467    4944 generic.go:356] "Generic (PLEG): container finished" podID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerID="9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6" exitCode=0
Nov 03 19:31:46.817037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.816552    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" event={"ID":"a9314ad0-e9cc-4aee-867f-2a70190ebbf0","Type":"ContainerDied","Data":"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"}
Nov 03 19:31:46.817037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.816663    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd" event={"ID":"a9314ad0-e9cc-4aee-867f-2a70190ebbf0","Type":"ContainerDied","Data":"e3bd6275ccfd87834f4bb88b87865d47f40299d76c58aa44b6725a6ca849f5c7"}
Nov 03 19:31:46.817037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.816756    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"
Nov 03 19:31:46.824669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.824645    4944 scope.go:122] "RemoveContainer" containerID="308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"
Nov 03 19:31:46.825163 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:46.825097    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90\": container with ID starting with 308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90 not found: ID does not exist" containerID="308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"
Nov 03 19:31:46.825281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.825173    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90"} err="failed to get container status \"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90\": rpc error: code = NotFound desc = could not find container \"308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90\": container with ID starting with 308f49e94ff207e584f2104b3fb317045e1959f98e2e238189b367c058261f90 not found: ID does not exist"
Nov 03 19:31:46.825281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.825193    4944 scope.go:122] "RemoveContainer" containerID="8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"
Nov 03 19:31:46.835485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.835468    4944 scope.go:122] "RemoveContainer" containerID="8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"
Nov 03 19:31:46.835735 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:46.835717    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503\": container with ID starting with 8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503 not found: ID does not exist" containerID="8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"
Nov 03 19:31:46.835814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.835740    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503"} err="failed to get container status \"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503\": rpc error: code = NotFound desc = could not find container \"8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503\": container with ID starting with 8528eb04ca8618a80efa141098dc74db74f7262c297e6bbb18a3f0c92fd08503 not found: ID does not exist"
Nov 03 19:31:46.835814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.835755    4944 scope.go:122] "RemoveContainer" containerID="3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"
Nov 03 19:31:46.844036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.844019    4944 scope.go:122] "RemoveContainer" containerID="3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"
Nov 03 19:31:46.844317 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:46.844297    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014\": container with ID starting with 3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014 not found: ID does not exist" containerID="3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"
Nov 03 19:31:46.844361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.844327    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014"} err="failed to get container status \"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014\": rpc error: code = NotFound desc = could not find container \"3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014\": container with ID starting with 3523deb339e2b1a36d7ffcc9206943c17f23bf50bb769c08217dd509ec093014 not found: ID does not exist"
Nov 03 19:31:46.844361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.844344    4944 scope.go:122] "RemoveContainer" containerID="9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"
Nov 03 19:31:46.852586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.852570    4944 scope.go:122] "RemoveContainer" containerID="9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"
Nov 03 19:31:46.852876 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:46.852856    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6\": container with ID starting with 9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6 not found: ID does not exist" containerID="9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"
Nov 03 19:31:46.852949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.852880    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6"} err="failed to get container status \"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6\": rpc error: code = NotFound desc = could not find container \"9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6\": container with ID starting with 9975418026d6e38f0ebd6f287f09386dcc76482fd02ae34ba7bfd21d37fc6ef6 not found: ID does not exist"
Nov 03 19:31:46.966930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.966898    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:31:46.971122 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:46.971091    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zdj72"]
Nov 03 19:31:47.071980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.071879    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:31:47.076234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.076205    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-4t7v2"]
Nov 03 19:31:47.182343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.182307    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 19:31:47.186334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.186305    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-7jmdj"]
Nov 03 19:31:47.289971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.289935    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:31:47.298893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.298842    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-lgzpd"]
Nov 03 19:31:47.941889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.941852    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" path="/var/lib/kubelet/pods/3910b36b-0adb-4567-b50c-a84b2784ce45/volumes"
Nov 03 19:31:47.942308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.942294    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" path="/var/lib/kubelet/pods/5edde6c1-e803-46f8-aa0a-689f4a2f7838/volumes"
Nov 03 19:31:47.942718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.942703    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" path="/var/lib/kubelet/pods/a9314ad0-e9cc-4aee-867f-2a70190ebbf0/volumes"
Nov 03 19:31:47.943421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:47.943406    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" path="/var/lib/kubelet/pods/f8a503d9-371d-4265-951c-2cc6b0abcf5c/volumes"
Nov 03 19:31:48.045726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:48.045691    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"]
Nov 03 19:31:48.049771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:48.049745    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"]
Nov 03 19:31:48.052008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:48.051988    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88d46164-w24zn"]
Nov 03 19:31:48.092230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:48.092199    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5bb0b7f-wc8zz"]
Nov 03 19:31:49.278222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:49.278185    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:49.304251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:49.304218    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing" containerID="cri-o://5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46" gracePeriod=25
Nov 03 19:31:49.464088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:49.464045    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing" containerID="cri-o://1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71" gracePeriod=25
Nov 03 19:31:49.933635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:49.933598    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e815a315-6d23-47c9-b353-520642ea23c5" path="/var/lib/kubelet/pods/e815a315-6d23-47c9-b353-520642ea23c5/volumes"
Nov 03 19:31:49.934168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:49.934151    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8759967-5c8d-4762-93d5-11137e28efc3" path="/var/lib/kubelet/pods/e8759967-5c8d-4762-93d5-11137e28efc3/volumes"
Nov 03 19:31:52.374183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.374139    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main" containerID="cri-o://e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9" gracePeriod=45
Nov 03 19:31:52.386322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.386278    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" containerID="cri-o://ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce" gracePeriod=45
Nov 03 19:31:52.390248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.390197    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main" containerID="cri-o://d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750" gracePeriod=45
Nov 03 19:31:52.390675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.390648    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main" containerID="cri-o://092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6" gracePeriod=45
Nov 03 19:31:52.708007 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.707978    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:31:52.710138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.710111    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:31:52.722177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.722154    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:31:52.730189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.730169    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:31:52.821011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.820976    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.821195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821027    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.821195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821058    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.821195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821097    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.821195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821141    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.821195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821179    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821209    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821254    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821299    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821352    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821380    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821400    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.821490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821388    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.821933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821583    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.821933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821593    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.821933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821601    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.821933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821634    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.821933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.821822    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.822185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822108    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\" (UniqueName: \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.822242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822153    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\") pod \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\" (UID: \"5ff60c1a-dbbf-41ad-b717-efa3eea672c2\") "
Nov 03 19:31:52.822242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822217    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.822343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822244    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.822343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822291    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\" (UniqueName: \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.822343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822327    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.822485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822355    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.822485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822394    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\" (UniqueName: \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\") pod \"9b997992-121e-47c8-ac54-d849d78fe5f5\" (UID: \"9b997992-121e-47c8-ac54-d849d78fe5f5\") "
Nov 03 19:31:52.822485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822434    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.822485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822459    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\") pod \"8290513e-d8eb-4c44-87cc-2e566616437b\" (UID: \"8290513e-d8eb-4c44-87cc-2e566616437b\") "
Nov 03 19:31:52.822658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822508    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\" (UniqueName: \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\") pod \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\" (UID: \"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e\") "
Nov 03 19:31:52.823065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.822994    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.823065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.823020    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.823065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.823034    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.823065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.823046    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.823065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.823059    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.824315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.823824    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824322    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824416    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824558    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824622    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824627    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.824856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824675    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.825040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824962    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.825040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.824976    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.827390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.827364    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.827645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.827611    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "kube-api-access-xp8cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:52.828842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.828808    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:52.828842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.828829    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:31:52.828965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.828884    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "kube-api-access-rtqx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:52.828965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.828885    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:52.829041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.829000    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs" pod "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" (UID: "b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:52.829342 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.829322    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs" pod "9b997992-121e-47c8-ac54-d849d78fe5f5" (UID: "9b997992-121e-47c8-ac54-d849d78fe5f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:52.830381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.830346    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm" pod "8290513e-d8eb-4c44-87cc-2e566616437b" (UID: "8290513e-d8eb-4c44-87cc-2e566616437b"). InnerVolumeSpecName "kube-api-access-9rdxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:52.830486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.830428    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr" pod "5ff60c1a-dbbf-41ad-b717-efa3eea672c2" (UID: "5ff60c1a-dbbf-41ad-b717-efa3eea672c2"). InnerVolumeSpecName "kube-api-access-mlttr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:52.861872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.861705    4944 generic.go:356] "Generic (PLEG): container finished" podID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerID="ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce" exitCode=0
Nov 03 19:31:52.861872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.861763    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"
Nov 03 19:31:52.861872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.861761    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" event={"ID":"5ff60c1a-dbbf-41ad-b717-efa3eea672c2","Type":"ContainerDied","Data":"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"}
Nov 03 19:31:52.861872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.861822    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" event={"ID":"5ff60c1a-dbbf-41ad-b717-efa3eea672c2","Type":"ContainerDied","Data":"e2a9a0f2fcd1e1b6ac1da7ebff667725fd46c2d6c6baa0a013f2b18fde11b4a1"}
Nov 03 19:31:52.861872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.861844    4944 scope.go:122] "RemoveContainer" containerID="ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"
Nov 03 19:31:52.863884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.863585    4944 generic.go:356] "Generic (PLEG): container finished" podID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerID="092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6" exitCode=0
Nov 03 19:31:52.863884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.863706    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" event={"ID":"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e","Type":"ContainerDied","Data":"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"}
Nov 03 19:31:52.863884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.863722    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"
Nov 03 19:31:52.863884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.863738    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd" event={"ID":"b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e","Type":"ContainerDied","Data":"0ecdb641e021b819241f03b452d0402a9b4e64c03870827d8fe6ecc66900d45e"}
Nov 03 19:31:52.866049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.866019    4944 generic.go:356] "Generic (PLEG): container finished" podID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerID="e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9" exitCode=0
Nov 03 19:31:52.866224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.866210    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"
Nov 03 19:31:52.866315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.866240    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" event={"ID":"9b997992-121e-47c8-ac54-d849d78fe5f5","Type":"ContainerDied","Data":"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"}
Nov 03 19:31:52.866315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.866274    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6" event={"ID":"9b997992-121e-47c8-ac54-d849d78fe5f5","Type":"ContainerDied","Data":"d9df37e06ca27b23c8b6b13181546b744c05c9b9f414f187198f7f522c214e28"}
Nov 03 19:31:52.868516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.868483    4944 generic.go:356] "Generic (PLEG): container finished" podID="8290513e-d8eb-4c44-87cc-2e566616437b" containerID="d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750" exitCode=0
Nov 03 19:31:52.868608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.868532    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" event={"ID":"8290513e-d8eb-4c44-87cc-2e566616437b","Type":"ContainerDied","Data":"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"}
Nov 03 19:31:52.868608 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.868558    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz" event={"ID":"8290513e-d8eb-4c44-87cc-2e566616437b","Type":"ContainerDied","Data":"ea5222d5f53581584635158bdf901744cad633382ebadbb545c10f944c79c852"}
Nov 03 19:31:52.868747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.868612    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"
Nov 03 19:31:52.881600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.881581    4944 scope.go:122] "RemoveContainer" containerID="ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"
Nov 03 19:31:52.884115 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:52.882257    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce\": container with ID starting with ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce not found: ID does not exist" containerID="ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"
Nov 03 19:31:52.884115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.882331    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce"} err="failed to get container status \"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce\": rpc error: code = NotFound desc = could not find container \"ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce\": container with ID starting with ed64fb025c1034d1e22868c64bd58f072a4631d74c658de59039a6bc265a61ce not found: ID does not exist"
Nov 03 19:31:52.884115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.882354    4944 scope.go:122] "RemoveContainer" containerID="092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"
Nov 03 19:31:52.893285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.893267    4944 scope.go:122] "RemoveContainer" containerID="092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"
Nov 03 19:31:52.893565 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:52.893536    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6\": container with ID starting with 092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6 not found: ID does not exist" containerID="092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"
Nov 03 19:31:52.893626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.893572    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6"} err="failed to get container status \"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6\": rpc error: code = NotFound desc = could not find container \"092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6\": container with ID starting with 092c12b8b5eaed44801d89701be3e95302697e6287f450b0d9170963b7ccf5e6 not found: ID does not exist"
Nov 03 19:31:52.893626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.893590    4944 scope.go:122] "RemoveContainer" containerID="e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"
Nov 03 19:31:52.902164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.902143    4944 scope.go:122] "RemoveContainer" containerID="e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"
Nov 03 19:31:52.902444 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:52.902424    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9\": container with ID starting with e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9 not found: ID does not exist" containerID="e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"
Nov 03 19:31:52.902519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.902455    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9"} err="failed to get container status \"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9\": rpc error: code = NotFound desc = could not find container \"e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9\": container with ID starting with e0fd8c05cf709ff5c587dc86a4ce74d410d8355596f52c99cf0c694b1d6485f9 not found: ID does not exist"
Nov 03 19:31:52.902519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.902479    4944 scope.go:122] "RemoveContainer" containerID="d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"
Nov 03 19:31:52.910486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.910470    4944 scope.go:122] "RemoveContainer" containerID="d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"
Nov 03 19:31:52.910744 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:31:52.910728    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750\": container with ID starting with d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750 not found: ID does not exist" containerID="d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"
Nov 03 19:31:52.910809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.910753    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750"} err="failed to get container status \"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750\": rpc error: code = NotFound desc = could not find container \"d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750\": container with ID starting with d28128d7b0dbcf6eb53e91b8ab4c77432424ec726f65db1914ae894021601750 not found: ID does not exist"
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929215    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929244    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929257    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b997992-121e-47c8-ac54-d849d78fe5f5-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929272    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929288    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlttr\" (UniqueName: \"kubernetes.io/projected/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-kube-api-access-mlttr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929304    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929317    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929333    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8290513e-d8eb-4c44-87cc-2e566616437b-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929346    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rdxm\" (UniqueName: \"kubernetes.io/projected/8290513e-d8eb-4c44-87cc-2e566616437b-kube-api-access-9rdxm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929360    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929375    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b997992-121e-47c8-ac54-d849d78fe5f5-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929391    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtqx7\" (UniqueName: \"kubernetes.io/projected/9b997992-121e-47c8-ac54-d849d78fe5f5-kube-api-access-rtqx7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929413    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929432    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929453    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8cp\" (UniqueName: \"kubernetes.io/projected/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-kube-api-access-xp8cp\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929473    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff60c1a-dbbf-41ad-b717-efa3eea672c2-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929493    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929511    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8290513e-d8eb-4c44-87cc-2e566616437b-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:52.929845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:52.929529    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:53.253645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.253614    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:31:53.257444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.257419    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-2-w1-010ed3-kserve-77bc7866f8-zlpb6"]
Nov 03 19:31:53.359876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.359837    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 19:31:53.365842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.365813    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/simulator-w1-010ed3-kserve-87467d74c-kn7rd"]
Nov 03 19:31:53.468215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.468182    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:31:53.472301 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.472277    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/premium-w1-010ed3-kserve-564c67c8b6-k8gwz"]
Nov 03 19:31:53.573657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.573593    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:31:53.577536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.577513    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv"]
Nov 03 19:31:53.650581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.650535    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-010ed3/distinct-w1-010ed3-kserve-6c866dfb4f-8vjsv" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:31:53.843878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.843778    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:53.933643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.933610    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" path="/var/lib/kubelet/pods/5ff60c1a-dbbf-41ad-b717-efa3eea672c2/volumes"
Nov 03 19:31:53.934129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.934113    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" path="/var/lib/kubelet/pods/8290513e-d8eb-4c44-87cc-2e566616437b/volumes"
Nov 03 19:31:53.934532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.934518    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" path="/var/lib/kubelet/pods/9b997992-121e-47c8-ac54-d849d78fe5f5/volumes"
Nov 03 19:31:53.935254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:53.935239    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" path="/var/lib/kubelet/pods/b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e/volumes"
Nov 03 19:31:54.174198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.174114    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:31:54.174374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.174117    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:31:54.424563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.424455    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"]
Nov 03 19:31:54.425027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425000    4944 cpu_manager.go:414] "containerMap: removing container" podUID="aba44825-3049-4613-a789-fe5931454a43" containerName="revoke-keys"
Nov 03 19:31:54.425027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425017    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba44825-3049-4613-a789-fe5931454a43" containerName="revoke-keys"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425031    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425038    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425049    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425054    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425062    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425067    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425076    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425081    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425088    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425093    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425118    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425123    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425130    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" containerName="payload-processing"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425134    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" containerName="payload-processing"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425144    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425148    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425159    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main"
Nov 03 19:31:54.425194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425163    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425228    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff60c1a-dbbf-41ad-b717-efa3eea672c2" containerName="main"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425240    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b997992-121e-47c8-ac54-d849d78fe5f5" containerName="main"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425250    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba44825-3049-4613-a789-fe5931454a43" containerName="revoke-keys"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425261    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8290513e-d8eb-4c44-87cc-2e566616437b" containerName="main"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425268    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edde6c1-e803-46f8-aa0a-689f4a2f7838" containerName="istio-proxy"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425273    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3910b36b-0adb-4567-b50c-a84b2784ce45" containerName="istio-proxy"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425279    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9314ad0-e9cc-4aee-867f-2a70190ebbf0" containerName="istio-proxy"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425285    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8a503d9-371d-4265-951c-2cc6b0abcf5c" containerName="istio-proxy"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425293    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04e78ee-97ec-4c4c-8cbb-f2f5fd112f1e" containerName="main"
Nov 03 19:31:54.426277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.425298    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f305f266-bb0c-473b-92a1-ebdb03c2f9d7" containerName="payload-processing"
Nov 03 19:31:54.429519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.429488    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.479478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.479416    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:31:54.480154 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.480133    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:31:54.482576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.482554    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"]
Nov 03 19:31:54.525548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.525524    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:31:54.543620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.543592    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.543757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.543643    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.543853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.543832    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fmk8\" (UniqueName: \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.579289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.579265    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:31:54.644845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.644818    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fmk8\" (UniqueName: \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.644967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.644876    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.644967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.644908    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.645536 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.645513    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.647121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.647098    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.701881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.701816    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fmk8\" (UniqueName: \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\") pod \"maas-api-revoke-sub-0d3e09b9-wgn4q\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:54.743110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:54.743084    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:55.111082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:55.111050    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"]
Nov 03 19:31:55.117121 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:31:55.117093    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d8e9d57_72e8_4552_a2a6_68f5ecaf7c0f.slice/crio-faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe WatchSource:0}: Error finding container faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe: Status 404 returned error can't find the container with id faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe
Nov 03 19:31:55.896925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:55.896887    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q" event={"ID":"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f","Type":"ContainerStarted","Data":"41ab8a7813ad722ed3cfef6ed6cd61cb75e86c91b49a7a15b6136b8d5cf1e388"}
Nov 03 19:31:55.897379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:55.896950    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q" event={"ID":"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f","Type":"ContainerStarted","Data":"faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe"}
Nov 03 19:31:56.005821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:56.005757    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q" podStartSLOduration=2.005739553 podStartE2EDuration="2.005739553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:31:56.004387086 +0000 UTC m=+2814.281740668" watchObservedRunningTime="2026-11-03 19:31:56.005739553 +0000 UTC m=+2814.283093135"
Nov 03 19:31:56.904729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:56.904686    4944 generic.go:356] "Generic (PLEG): container finished" podID="7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" containerID="41ab8a7813ad722ed3cfef6ed6cd61cb75e86c91b49a7a15b6136b8d5cf1e388" exitCode=0
Nov 03 19:31:56.905198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:56.904771    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q" event={"ID":"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f","Type":"ContainerDied","Data":"41ab8a7813ad722ed3cfef6ed6cd61cb75e86c91b49a7a15b6136b8d5cf1e388"}
Nov 03 19:31:58.061319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.061290    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:58.180606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.180528    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\" (UniqueName: \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\") pod \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") "
Nov 03 19:31:58.180606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.180573    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\") pod \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") "
Nov 03 19:31:58.180888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.180745    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\") pod \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\" (UID: \"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f\") "
Nov 03 19:31:58.181176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.181146    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca" pod "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" (UID: "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:31:58.182737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.182711    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca" pod "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" (UID: "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:31:58.182885 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.182832    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8" pod "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" (UID: "7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f"). InnerVolumeSpecName "kube-api-access-5fmk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:31:58.281807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.281686    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:58.281986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.281823    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fmk8\" (UniqueName: \"kubernetes.io/projected/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-kube-api-access-5fmk8\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:58.281986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.281844    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:31:58.453974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.453888    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:31:58.918058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.918027    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"
Nov 03 19:31:58.918250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.918061    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q" event={"ID":"7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f","Type":"ContainerDied","Data":"faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe"}
Nov 03 19:31:58.918250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:31:58.918113    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faadebb8d6793591431929441271042d03252a5270155a3c234a33e054f66bfe"
Nov 03 19:32:03.304729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:03.304685    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:03.551866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:03.551831    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:03.899257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:03.899221    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:04.174677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.174582    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:04.174677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.174640    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:04.317570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.317536    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:32:04.318126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.318110    4944 cpu_manager.go:414] "containerMap: removing container" podUID="7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" containerName="revoke-keys"
Nov 03 19:32:04.318199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.318127    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" containerName="revoke-keys"
Nov 03 19:32:04.318239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.318230    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" containerName="revoke-keys"
Nov 03 19:32:04.320785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.320764    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.372515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.372322    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-24cdad-openshift-default-dockercfg-8d2ft\""
Nov 03 19:32:04.421457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.421424    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:32:04.445420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445327    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445372    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445408    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445540    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445599    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445733    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445844    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445878    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.445945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.445928    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.519436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.519401    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:04.547068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547029    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.547231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547080    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.547231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547118    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.547231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547169    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.547231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547211    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.547452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547280    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.548084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547742    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.548084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547964    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.548084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.547364    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.548446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.548419    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.548654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.548634    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.549209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.549188    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.549348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.548277    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.549714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.548299    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.553816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.550594    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.556530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.556501    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.604639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.604607    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.605845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.605780    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\") pod \"e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") " pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.633721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.633693    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:04.868954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:04.868919    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:05.057328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.057293    4944 scope.go:122] "RemoveContainer" containerID="cc91d03ba24b9ad42d9bd8d720973633c3e9eeba46ea1bad2084d32238fc3eb2"
Nov 03 19:32:05.058708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.058683    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:32:05.063575 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:05.063549    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda23466c0_a35e_4646_b152_859d188fa066.slice/crio-87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c WatchSource:0}: Error finding container 87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c: Status 404 returned error can't find the container with id 87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c
Nov 03 19:32:05.066702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.066686    4944 scope.go:122] "RemoveContainer" containerID="3313df217ed57026447d054f9ca25ec429929dd6502b1649f34bda703d525dc5"
Nov 03 19:32:05.067048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.067013    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:32:05.067109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.067085    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:32:05.067160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.067119    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:32:05.075574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.075548    4944 scope.go:122] "RemoveContainer" containerID="86b556381f783dc9b078ef0ae434b47b4c81e03e3a8a918b7821222fd57cbfa5"
Nov 03 19:32:05.227780 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:32:05.227733    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:32:05.959973 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.959935    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerStarted","Data":"966b288281c61d6cb01a21e9b2576cd242ca2ed78c301507b8397879e1663811"}
Nov 03 19:32:05.960410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:05.959987    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerStarted","Data":"87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c"}
Nov 03 19:32:06.166985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.166924    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podStartSLOduration=2.166906691 podStartE2EDuration="2.166906691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:06.11928734 +0000 UTC m=+2824.396640923" watchObservedRunningTime="2026-11-03 19:32:06.166906691 +0000 UTC m=+2824.444260275"
Nov 03 19:32:06.479952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.479921    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:06.634651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.634620    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:06.640001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.639977    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:06.968416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.968381    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:06.969401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:06.969379    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:32:07.318636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.318601    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"]
Nov 03 19:32:07.322685 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.322667    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.373124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.373092    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:32:07.373412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.373391    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:32:07.373624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.373602    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:32:07.373911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.373895    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:32:07.377486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.377464    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"]
Nov 03 19:32:07.481774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.481733    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.481983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.481779    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgj9q\" (UniqueName: \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.482026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.481990    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.583367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.583281    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.583367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.583355    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.583575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.583396    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgj9q\" (UniqueName: \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.584068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.584044    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.585676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.585653    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.640823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.640778    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgj9q\" (UniqueName: \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\") pod \"maas-api-revoke-sub-8709ada9-qjwjp\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:07.935071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:07.934989    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:08.319242 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:08.319200    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"]
Nov 03 19:32:08.321494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:08.321458    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:08.324834 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:08.324810    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode309f572_cbdb_4354_bcbb_048125161fda.slice/crio-987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74 WatchSource:0}: Error finding container 987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74: Status 404 returned error can't find the container with id 987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74
Nov 03 19:32:08.968864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:08.968765    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:08.981943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:08.981903    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp" event={"ID":"e309f572-cbdb-4354-bcbb-048125161fda","Type":"ContainerStarted","Data":"b0049d8a334877b8e31538081d9eb29f9fdc0d2e9f8f65d6dbedcade6cc7d602"}
Nov 03 19:32:08.982102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:08.982010    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp" event={"ID":"e309f572-cbdb-4354-bcbb-048125161fda","Type":"ContainerStarted","Data":"987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74"}
Nov 03 19:32:09.097418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.097364    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp" podStartSLOduration=2.097345921 podStartE2EDuration="2.097345921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:09.096129278 +0000 UTC m=+2827.373482874" watchObservedRunningTime="2026-11-03 19:32:09.097345921 +0000 UTC m=+2827.374699505"
Nov 03 19:32:09.301095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.301066    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:09.575772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.575750    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 19:32:09.708850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.708819    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\") pod \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") "
Nov 03 19:32:09.709035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.709016    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\" (UniqueName: \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\") pod \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\" (UID: \"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc\") "
Nov 03 19:32:09.709153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.709132    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume" pod "81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" (UID: "81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:09.709479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.709457    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:09.711082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.711054    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj" pod "81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" (UID: "81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc"). InnerVolumeSpecName "kube-api-access-7wgkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:09.726251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.726233    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 19:32:09.810268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.810233    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgkj\" (UniqueName: \"kubernetes.io/projected/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc-kube-api-access-7wgkj\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:09.911258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.911169    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\" (UniqueName: \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\") pod \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") "
Nov 03 19:32:09.911258 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.911241    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\") pod \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\" (UID: \"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e\") "
Nov 03 19:32:09.911716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.911692    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume" pod "1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" (UID: "1f4f7cde-10c0-4e15-86e4-de4cf5d6675e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:09.913488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.913459    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2" pod "1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" (UID: "1f4f7cde-10c0-4e15-86e4-de4cf5d6675e"). InnerVolumeSpecName "kube-api-access-zkrd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:09.988471 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.988437    4944 generic.go:356] "Generic (PLEG): container finished" podID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerID="1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71" exitCode=0
Nov 03 19:32:09.988594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.988487    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"
Nov 03 19:32:09.988594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.988523    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" event={"ID":"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e","Type":"ContainerDied","Data":"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"}
Nov 03 19:32:09.988594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.988582    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs" event={"ID":"1f4f7cde-10c0-4e15-86e4-de4cf5d6675e","Type":"ContainerDied","Data":"e8891451b0cc51a6cea027424f3b31644e550f85e16da7029ce2a25fa0a44061"}
Nov 03 19:32:09.988819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.988612    4944 scope.go:122] "RemoveContainer" containerID="1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"
Nov 03 19:32:09.990049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.990024    4944 generic.go:356] "Generic (PLEG): container finished" podID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerID="5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46" exitCode=0
Nov 03 19:32:09.990151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.990068    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"
Nov 03 19:32:09.990151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.990114    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" event={"ID":"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc","Type":"ContainerDied","Data":"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"}
Nov 03 19:32:09.990295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.990166    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv" event={"ID":"81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc","Type":"ContainerDied","Data":"43b6d8bd3e8e8e2e5d3fac60c39ef3d4cf3573bd35432941da39e65fd0bd9b68"}
Nov 03 19:32:09.992171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.992147    4944 generic.go:356] "Generic (PLEG): container finished" podID="e309f572-cbdb-4354-bcbb-048125161fda" containerID="b0049d8a334877b8e31538081d9eb29f9fdc0d2e9f8f65d6dbedcade6cc7d602" exitCode=0
Nov 03 19:32:09.992285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:09.992205    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp" event={"ID":"e309f572-cbdb-4354-bcbb-048125161fda","Type":"ContainerDied","Data":"b0049d8a334877b8e31538081d9eb29f9fdc0d2e9f8f65d6dbedcade6cc7d602"}
Nov 03 19:32:10.001109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.001090    4944 scope.go:122] "RemoveContainer" containerID="1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"
Nov 03 19:32:10.001410 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:32:10.001392    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71\": container with ID starting with 1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71 not found: ID does not exist" containerID="1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"
Nov 03 19:32:10.001482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.001422    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71"} err="failed to get container status \"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71\": rpc error: code = NotFound desc = could not find container \"1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71\": container with ID starting with 1a177aada372f0d4f9ab60289fc3df9b613f6c2461bac861b18c9e41a542ff71 not found: ID does not exist"
Nov 03 19:32:10.001482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.001446    4944 scope.go:122] "RemoveContainer" containerID="5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"
Nov 03 19:32:10.009589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.009573    4944 scope.go:122] "RemoveContainer" containerID="5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"
Nov 03 19:32:10.009862 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:32:10.009842    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46\": container with ID starting with 5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46 not found: ID does not exist" containerID="5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"
Nov 03 19:32:10.009969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.009867    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46"} err="failed to get container status \"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46\": rpc error: code = NotFound desc = could not find container \"5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46\": container with ID starting with 5fda465ac86b5ee69ae6e95f458001eb30c2c3e083220afe6ac458c5f6e87e46 not found: ID does not exist"
Nov 03 19:32:10.012448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.012428    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkrd2\" (UniqueName: \"kubernetes.io/projected/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-kube-api-access-zkrd2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:10.012502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.012451    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:10.141772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.141737    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 19:32:10.145716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.145691    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-010ed3-5f894d94c4-kbvbv"]
Nov 03 19:32:10.356641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.356610    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 19:32:10.362059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:10.362033    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-010ed3-cb7f7b747-drgzs"]
Nov 03 19:32:11.136742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.136718    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:11.325966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.325879    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\") pod \"e309f572-cbdb-4354-bcbb-048125161fda\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") "
Nov 03 19:32:11.326127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.326040    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\") pod \"e309f572-cbdb-4354-bcbb-048125161fda\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") "
Nov 03 19:32:11.326127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.326116    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\" (UniqueName: \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\") pod \"e309f572-cbdb-4354-bcbb-048125161fda\" (UID: \"e309f572-cbdb-4354-bcbb-048125161fda\") "
Nov 03 19:32:11.326417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.326379    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca" pod "e309f572-cbdb-4354-bcbb-048125161fda" (UID: "e309f572-cbdb-4354-bcbb-048125161fda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:11.326559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.326540    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:11.327957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.327934    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca" pod "e309f572-cbdb-4354-bcbb-048125161fda" (UID: "e309f572-cbdb-4354-bcbb-048125161fda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:11.328188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.328160    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q" pod "e309f572-cbdb-4354-bcbb-048125161fda" (UID: "e309f572-cbdb-4354-bcbb-048125161fda"). InnerVolumeSpecName "kube-api-access-hgj9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:11.427760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.427729    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e309f572-cbdb-4354-bcbb-048125161fda-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:11.427760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.427756    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgj9q\" (UniqueName: \"kubernetes.io/projected/e309f572-cbdb-4354-bcbb-048125161fda-kube-api-access-hgj9q\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:11.934104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.934069    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" path="/var/lib/kubelet/pods/1f4f7cde-10c0-4e15-86e4-de4cf5d6675e/volumes"
Nov 03 19:32:11.934472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:11.934453    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" path="/var/lib/kubelet/pods/81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc/volumes"
Nov 03 19:32:12.006546 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.006511    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp" event={"ID":"e309f572-cbdb-4354-bcbb-048125161fda","Type":"ContainerDied","Data":"987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74"}
Nov 03 19:32:12.006692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.006558    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="987df09a8d035d4c7dc73c5dc488ad1bc0bf8ad62ff0637edaa63de87cbbdd74"
Nov 03 19:32:12.006692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.006521    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"
Nov 03 19:32:12.195643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.195557    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:12.448363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.448279    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:12.494554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.494519    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:32:12.495008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.494993    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e309f572-cbdb-4354-bcbb-048125161fda" containerName="revoke-keys"
Nov 03 19:32:12.495066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495009    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e309f572-cbdb-4354-bcbb-048125161fda" containerName="revoke-keys"
Nov 03 19:32:12.495066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495024    4944 cpu_manager.go:414] "containerMap: removing container" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing"
Nov 03 19:32:12.495066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495031    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing"
Nov 03 19:32:12.495066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495045    4944 cpu_manager.go:414] "containerMap: removing container" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing"
Nov 03 19:32:12.495066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495051    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing"
Nov 03 19:32:12.495241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495149    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4f7cde-10c0-4e15-86e4-de4cf5d6675e" containerName="payload-pre-processing"
Nov 03 19:32:12.495241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495167    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e309f572-cbdb-4354-bcbb-048125161fda" containerName="revoke-keys"
Nov 03 19:32:12.495241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.495173    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d0aa4d-e2ea-4a87-bcb6-11ad5826dafc" containerName="payload-processing"
Nov 03 19:32:12.499076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.499055    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.537966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.537936    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv8h\" (UniqueName: \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.538120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.538004    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.549527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.549490    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24cdad\""
Nov 03 19:32:12.550312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.550285    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24cdad-dockercfg-6mvpd\""
Nov 03 19:32:12.599866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.599823    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:32:12.639160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.639125    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv8h\" (UniqueName: \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.639318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.639212    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.639981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.639963    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.656997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.656967    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:32:12.660235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.660216    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.696351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.696329    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv8h\" (UniqueName: \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\") pod \"payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.715676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.715619    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:32:12.733911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.733883    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"]
Nov 03 19:32:12.737049 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.737030    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.740922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.740893    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.741050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.741029    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.741103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.741066    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhbwn\" (UniqueName: \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.741163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.741100    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvw2\" (UniqueName: \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.741163 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.741125    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.788717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.788679    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:32:12.788888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.788745    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:32:12.788888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.788753    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:32:12.788888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.788776    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:32:12.792197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.792175    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"]
Nov 03 19:32:12.795946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.795924    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"]
Nov 03 19:32:12.799754 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.799735    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.811710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.811682    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:12.842412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842380    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.842560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842431    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.842560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842547    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.842664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842632    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhbwn\" (UniqueName: \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.842722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842688    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvw2\" (UniqueName: \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.842776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842719    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.842883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842771    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.842942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.842898    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf57r\" (UniqueName: \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.843321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.843299    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.843668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.843646    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.844918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.844893    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.857179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.857144    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"]
Nov 03 19:32:12.901436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.900869    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvw2\" (UniqueName: \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\") pod \"maas-api-revoke-sub-6a542e0a-rdlz9\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:12.901436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.901386    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhbwn\" (UniqueName: \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\") pod \"payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:12.944613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.944575    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.944833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.944688    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf57r\" (UniqueName: \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.944833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.944749    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.945305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.945284    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.946998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.946981    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:12.972853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:12.972783    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:13.002181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.002155    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf57r\" (UniqueName: \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\") pod \"maas-api-revoke-sub-30b86be4-r5gkj\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:13.050413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.050179    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:13.051509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.051481    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:13.114999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.114724    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:13.198219 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:13.198184    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a780f3_f7ae_4499_ba3e_ff4e7802d425.slice/crio-83a25fe98d7c01bc87e6a8697edaf62febd7a3a83c8e33239faf088330c41c3d WatchSource:0}: Error finding container 83a25fe98d7c01bc87e6a8697edaf62febd7a3a83c8e33239faf088330c41c3d: Status 404 returned error can't find the container with id 83a25fe98d7c01bc87e6a8697edaf62febd7a3a83c8e33239faf088330c41c3d
Nov 03 19:32:13.198920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.198892    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:32:13.348021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.347992    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:32:13.353256 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:13.353227    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65979675_0926_4ce6_a17c_62db3d89992d.slice/crio-54ecd94cc010e43b58b1726b35948a977571bbf1c5912d8e968e410ec944c6a0 WatchSource:0}: Error finding container 54ecd94cc010e43b58b1726b35948a977571bbf1c5912d8e968e410ec944c6a0: Status 404 returned error can't find the container with id 54ecd94cc010e43b58b1726b35948a977571bbf1c5912d8e968e410ec944c6a0
Nov 03 19:32:13.434814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.434761    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"]
Nov 03 19:32:13.440979 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:13.440943    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeae44a7_460c_4ed3_afe9_726bd8050d07.slice/crio-29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d WatchSource:0}: Error finding container 29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d: Status 404 returned error can't find the container with id 29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d
Nov 03 19:32:13.497522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:13.497497    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"]
Nov 03 19:32:13.502382 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:13.502358    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08f3bb5_1384_4ddd_bf0b_7b9f371a9b31.slice/crio-47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233 WatchSource:0}: Error finding container 47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233: Status 404 returned error can't find the container with id 47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233
Nov 03 19:32:14.023876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.023733    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9" event={"ID":"feae44a7-460c-4ed3-afe9-726bd8050d07","Type":"ContainerStarted","Data":"c3e47056a8e9ebcb6cf16205d85170c6ec0be8a62397f5ba8503b635395ced90"}
Nov 03 19:32:14.023876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.023860    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9" event={"ID":"feae44a7-460c-4ed3-afe9-726bd8050d07","Type":"ContainerStarted","Data":"29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d"}
Nov 03 19:32:14.026171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.026140    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" event={"ID":"65979675-0926-4ce6-a17c-62db3d89992d","Type":"ContainerStarted","Data":"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"}
Nov 03 19:32:14.026308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.026197    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" event={"ID":"65979675-0926-4ce6-a17c-62db3d89992d","Type":"ContainerStarted","Data":"54ecd94cc010e43b58b1726b35948a977571bbf1c5912d8e968e410ec944c6a0"}
Nov 03 19:32:14.026308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.026246    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:14.028119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.028091    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj" event={"ID":"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31","Type":"ContainerStarted","Data":"939e609916ed0649392837b5a5def8170228b5deaa4e999c44c0d0987fe4def0"}
Nov 03 19:32:14.028266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.028146    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj" event={"ID":"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31","Type":"ContainerStarted","Data":"47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233"}
Nov 03 19:32:14.030026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.030001    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" event={"ID":"f2a780f3-f7ae-4499-ba3e-ff4e7802d425","Type":"ContainerStarted","Data":"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"}
Nov 03 19:32:14.030171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.030044    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" event={"ID":"f2a780f3-f7ae-4499-ba3e-ff4e7802d425","Type":"ContainerStarted","Data":"83a25fe98d7c01bc87e6a8697edaf62febd7a3a83c8e33239faf088330c41c3d"}
Nov 03 19:32:14.030171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.030087    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:14.133965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.133850    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9" podStartSLOduration=2.13383607 podStartE2EDuration="2.13383607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:14.132980148 +0000 UTC m=+2832.410333731" watchObservedRunningTime="2026-11-03 19:32:14.13383607 +0000 UTC m=+2832.411189654"
Nov 03 19:32:14.168572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.168541    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.241531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.241475    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" podStartSLOduration=2.241461187 podStartE2EDuration="2.241461187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:14.240722546 +0000 UTC m=+2832.518076130" watchObservedRunningTime="2026-11-03 19:32:14.241461187 +0000 UTC m=+2832.518814797"
Nov 03 19:32:14.350368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.350314    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj" podStartSLOduration=2.350297368 podStartE2EDuration="2.350297368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:14.348026022 +0000 UTC m=+2832.625379605" watchObservedRunningTime="2026-11-03 19:32:14.350297368 +0000 UTC m=+2832.627650997"
Nov 03 19:32:14.415869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.415776    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.457105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.457051    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" podStartSLOduration=2.457037205 podStartE2EDuration="2.457037205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:32:14.455709072 +0000 UTC m=+2832.733062680" watchObservedRunningTime="2026-11-03 19:32:14.457037205 +0000 UTC m=+2832.734390788"
Nov 03 19:32:14.705572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.705484    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:14.969739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:14.969657    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:15.041060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.041026    4944 generic.go:356] "Generic (PLEG): container finished" podID="d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" containerID="939e609916ed0649392837b5a5def8170228b5deaa4e999c44c0d0987fe4def0" exitCode=0
Nov 03 19:32:15.041434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.041406    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj" event={"ID":"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31","Type":"ContainerDied","Data":"939e609916ed0649392837b5a5def8170228b5deaa4e999c44c0d0987fe4def0"}
Nov 03 19:32:15.043307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.043280    4944 generic.go:356] "Generic (PLEG): container finished" podID="feae44a7-460c-4ed3-afe9-726bd8050d07" containerID="c3e47056a8e9ebcb6cf16205d85170c6ec0be8a62397f5ba8503b635395ced90" exitCode=0
Nov 03 19:32:15.043418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.043372    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9" event={"ID":"feae44a7-460c-4ed3-afe9-726bd8050d07","Type":"ContainerDied","Data":"c3e47056a8e9ebcb6cf16205d85170c6ec0be8a62397f5ba8503b635395ced90"}
Nov 03 19:32:15.081236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.081203    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"]
Nov 03 19:32:15.085673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.085650    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c3e530-8dhkc"]
Nov 03 19:32:15.215128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.215093    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:15.558904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.558852    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:15.822527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.822447    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:15.933955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:15.933922    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c740017-0fbd-4a8b-b75a-fca050322793" path="/var/lib/kubelet/pods/5c740017-0fbd-4a8b-b75a-fca050322793/volumes"
Nov 03 19:32:16.201771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.201698    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:16.230812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.230768    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:16.280189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280159    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\") pod \"feae44a7-460c-4ed3-afe9-726bd8050d07\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") "
Nov 03 19:32:16.280348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280213    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\" (UniqueName: \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\") pod \"feae44a7-460c-4ed3-afe9-726bd8050d07\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") "
Nov 03 19:32:16.280348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280259    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\") pod \"feae44a7-460c-4ed3-afe9-726bd8050d07\" (UID: \"feae44a7-460c-4ed3-afe9-726bd8050d07\") "
Nov 03 19:32:16.280348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280335    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\" (UniqueName: \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\") pod \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") "
Nov 03 19:32:16.280537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280388    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\") pod \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") "
Nov 03 19:32:16.280537 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280438    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\") pod \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\" (UID: \"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31\") "
Nov 03 19:32:16.280673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280538    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca" pod "feae44a7-460c-4ed3-afe9-726bd8050d07" (UID: "feae44a7-460c-4ed3-afe9-726bd8050d07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:16.280840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280815    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:16.280960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.280935    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca" pod "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" (UID: "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:16.282458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.282432    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca" pod "feae44a7-460c-4ed3-afe9-726bd8050d07" (UID: "feae44a7-460c-4ed3-afe9-726bd8050d07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:16.282564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.282511    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2" pod "feae44a7-460c-4ed3-afe9-726bd8050d07" (UID: "feae44a7-460c-4ed3-afe9-726bd8050d07"). InnerVolumeSpecName "kube-api-access-9tvw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:16.282564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.282518    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca" pod "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" (UID: "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:16.282564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.282543    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r" pod "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" (UID: "d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31"). InnerVolumeSpecName "kube-api-access-hf57r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:16.382410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.382333    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:16.382410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.382362    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvw2\" (UniqueName: \"kubernetes.io/projected/feae44a7-460c-4ed3-afe9-726bd8050d07-kube-api-access-9tvw2\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:16.382410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.382372    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/feae44a7-460c-4ed3-afe9-726bd8050d07-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:16.382410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.382382    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf57r\" (UniqueName: \"kubernetes.io/projected/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-kube-api-access-hf57r\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:16.382410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:16.382393    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:17.056725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.056687    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"
Nov 03 19:32:17.056725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.056711    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj" event={"ID":"d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31","Type":"ContainerDied","Data":"47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233"}
Nov 03 19:32:17.057217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.056754    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47b0e95eba3f88e729c4368747806fecaebb22d905db644a17adb7b9f49e1233"
Nov 03 19:32:17.058151 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.058129    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"
Nov 03 19:32:17.058275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.058146    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9" event={"ID":"feae44a7-460c-4ed3-afe9-726bd8050d07","Type":"ContainerDied","Data":"29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d"}
Nov 03 19:32:17.058275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:17.058190    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e19d2f9be7871e0145242e7c6a6b15c0c94a995416eb98fe8756cc7cb4e05d"
Nov 03 19:32:20.000224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:20.000176    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:21.758743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:21.758700    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:21.962072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:21.962039    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:32:21.962271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:21.962230    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy" containerID="cri-o://ba9f7a800b031b9926e535c13c132f7e03bf2a527bec062d1bb9e25394e077fa" gracePeriod=30
Nov 03 19:32:22.010349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.010260    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:22.264540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.264466    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:22.553463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.553387    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"]
Nov 03 19:32:22.553898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.553884    4944 cpu_manager.go:414] "containerMap: removing container" podUID="feae44a7-460c-4ed3-afe9-726bd8050d07" containerName="revoke-keys"
Nov 03 19:32:22.553941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.553900    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feae44a7-460c-4ed3-afe9-726bd8050d07" containerName="revoke-keys"
Nov 03 19:32:22.553981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.553940    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" containerName="revoke-keys"
Nov 03 19:32:22.553981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.553946    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" containerName="revoke-keys"
Nov 03 19:32:22.554047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.554017    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" containerName="revoke-keys"
Nov 03 19:32:22.554047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.554027    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="feae44a7-460c-4ed3-afe9-726bd8050d07" containerName="revoke-keys"
Nov 03 19:32:22.556833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.556813    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.608586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.608548    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:32:22.608586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.608571    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:32:22.608861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.608601    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:32:22.608861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.608604    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:32:22.611758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.611731    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"]
Nov 03 19:32:22.649253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.649222    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.649378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.649273    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.649434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.649396    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnbvl\" (UniqueName: \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.750667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.750634    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.750835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.750691    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.750835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.750752    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnbvl\" (UniqueName: \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.751308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.751290    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.753023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.753003    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.807645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.807582    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnbvl\" (UniqueName: \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:22.869776 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:22.869744    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:23.138881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.138740    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:23.239092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.239059    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"]
Nov 03 19:32:23.244748 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:23.244718    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b03ce2_5894_42c8_b87c_462330e01902.slice/crio-0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57 WatchSource:0}: Error finding container 0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57: Status 404 returned error can't find the container with id 0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57
Nov 03 19:32:23.373469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.373439    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"]
Nov 03 19:32:23.377430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.377409    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.431577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.431548    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"]
Nov 03 19:32:23.457772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.457740    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.457948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.457821    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.458016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.457953    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mkb7\" (UniqueName: \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.559133 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.559104    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.559273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.559172    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mkb7\" (UniqueName: \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.559345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.559288    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.559817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.559767    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.561484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.561463    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.616096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.616073    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mkb7\" (UniqueName: \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\") pod \"maas-api-revoke-keys-e2e-inf-a-73a216-bg766\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:23.690486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:23.690414    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:24.057295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.057262    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"]
Nov 03 19:32:24.062910 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:24.062877    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6030ac_6021_4b5f_aef5_1dbcfe683381.slice/crio-c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56 WatchSource:0}: Error finding container c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56: Status 404 returned error can't find the container with id c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56
Nov 03 19:32:24.106627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.106602    4944 generic.go:356] "Generic (PLEG): container finished" podID="34b03ce2-5894-42c8-b87c-462330e01902" containerID="44ee90ecbc629608caab63f25cc9f6737c0ecb9cb57e7c637c59a33acaf578f2" exitCode=0
Nov 03 19:32:24.106735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.106679    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df" event={"ID":"34b03ce2-5894-42c8-b87c-462330e01902","Type":"ContainerDied","Data":"44ee90ecbc629608caab63f25cc9f6737c0ecb9cb57e7c637c59a33acaf578f2"}
Nov 03 19:32:24.106735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.106732    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df" event={"ID":"34b03ce2-5894-42c8-b87c-462330e01902","Type":"ContainerStarted","Data":"0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57"}
Nov 03 19:32:24.108033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.108009    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766" event={"ID":"0d6030ac-6021-4b5f-aef5-1dbcfe683381","Type":"ContainerStarted","Data":"c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56"}
Nov 03 19:32:24.933660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:24.933625    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:25.046556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.046530    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:32:25.046718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.046597    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:32:25.115009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.114975    4944 generic.go:356] "Generic (PLEG): container finished" podID="0d6030ac-6021-4b5f-aef5-1dbcfe683381" containerID="48a5f24c8938ba11b35660077356a06e5a1e556135aba732915d353c6fe3070c" exitCode=0
Nov 03 19:32:25.115413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.115038    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766" event={"ID":"0d6030ac-6021-4b5f-aef5-1dbcfe683381","Type":"ContainerDied","Data":"48a5f24c8938ba11b35660077356a06e5a1e556135aba732915d353c6fe3070c"}
Nov 03 19:32:25.283205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.283179    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:25.380384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.380346    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\" (UniqueName: \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\") pod \"34b03ce2-5894-42c8-b87c-462330e01902\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") "
Nov 03 19:32:25.380646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.380592    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\") pod \"34b03ce2-5894-42c8-b87c-462330e01902\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") "
Nov 03 19:32:25.380757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.380691    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\") pod \"34b03ce2-5894-42c8-b87c-462330e01902\" (UID: \"34b03ce2-5894-42c8-b87c-462330e01902\") "
Nov 03 19:32:25.380956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.380932    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca" pod "34b03ce2-5894-42c8-b87c-462330e01902" (UID: "34b03ce2-5894-42c8-b87c-462330e01902"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:25.381100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.381081    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:25.382490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.382467    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl" pod "34b03ce2-5894-42c8-b87c-462330e01902" (UID: "34b03ce2-5894-42c8-b87c-462330e01902"). InnerVolumeSpecName "kube-api-access-qnbvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:25.382678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.382658    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca" pod "34b03ce2-5894-42c8-b87c-462330e01902" (UID: "34b03ce2-5894-42c8-b87c-462330e01902"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:25.482583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.482485    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b03ce2-5894-42c8-b87c-462330e01902-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:25.482583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:25.482520    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnbvl\" (UniqueName: \"kubernetes.io/projected/34b03ce2-5894-42c8-b87c-462330e01902-kube-api-access-qnbvl\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:26.122365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.122333    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"
Nov 03 19:32:26.122763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.122367    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df" event={"ID":"34b03ce2-5894-42c8-b87c-462330e01902","Type":"ContainerDied","Data":"0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57"}
Nov 03 19:32:26.122763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.122416    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0670a6ec3a64664e0a9f42d871ed517281bdf68ca57b775f6022e17d6acccb57"
Nov 03 19:32:26.262765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.262742    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:26.390417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.390382    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\") pod \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") "
Nov 03 19:32:26.390580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.390450    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\" (UniqueName: \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\") pod \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") "
Nov 03 19:32:26.390627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.390604    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\") pod \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\" (UID: \"0d6030ac-6021-4b5f-aef5-1dbcfe683381\") "
Nov 03 19:32:26.390854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.390826    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca" pod "0d6030ac-6021-4b5f-aef5-1dbcfe683381" (UID: "0d6030ac-6021-4b5f-aef5-1dbcfe683381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:26.391074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.391048    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:26.392587 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.392559    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca" pod "0d6030ac-6021-4b5f-aef5-1dbcfe683381" (UID: "0d6030ac-6021-4b5f-aef5-1dbcfe683381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:32:26.392719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.392684    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7" pod "0d6030ac-6021-4b5f-aef5-1dbcfe683381" (UID: "0d6030ac-6021-4b5f-aef5-1dbcfe683381"). InnerVolumeSpecName "kube-api-access-5mkb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:26.492153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.492130    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d6030ac-6021-4b5f-aef5-1dbcfe683381-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:26.492285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:26.492155    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mkb7\" (UniqueName: \"kubernetes.io/projected/0d6030ac-6021-4b5f-aef5-1dbcfe683381-kube-api-access-5mkb7\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.129664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.129633    4944 generic.go:356] "Generic (PLEG): container finished" podID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerID="ba9f7a800b031b9926e535c13c132f7e03bf2a527bec062d1bb9e25394e077fa" exitCode=0
Nov 03 19:32:27.130171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.129705    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" event={"ID":"2b728339-4a94-4aa4-b75f-c1b16589d9ce","Type":"ContainerDied","Data":"ba9f7a800b031b9926e535c13c132f7e03bf2a527bec062d1bb9e25394e077fa"}
Nov 03 19:32:27.131130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.131106    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"
Nov 03 19:32:27.131249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.131141    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766" event={"ID":"0d6030ac-6021-4b5f-aef5-1dbcfe683381","Type":"ContainerDied","Data":"c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56"}
Nov 03 19:32:27.131249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.131181    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30691b5a17f79a121dec6b01091215ad25e2d8aafec4ac9839ce4a2add9cc56"
Nov 03 19:32:27.210087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.210062    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:32:27.301430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301397    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301444    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301468    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301498    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301563    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301718    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301824    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301854    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\") pod \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\" (UID: \"2b728339-4a94-4aa4-b75f-c1b16589d9ce\") "
Nov 03 19:32:27.301904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301887    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:27.302190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.301971    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:27.302190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302122    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:27.302190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302161    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:27.302360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302252    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:27.302506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302484    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.302580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302513    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.302580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302529    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.302580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302543    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.302580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.302558    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.303736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.303712    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:32:27.303868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.303718    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:27.303868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.303842    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "kube-api-access-mfg4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:27.303958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.303912    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy" pod "2b728339-4a94-4aa4-b75f-c1b16589d9ce" (UID: "2b728339-4a94-4aa4-b75f-c1b16589d9ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:32:27.403734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.403697    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.403734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.403737    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.403959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.403753    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfg4v\" (UniqueName: \"kubernetes.io/projected/2b728339-4a94-4aa4-b75f-c1b16589d9ce-kube-api-access-mfg4v\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:27.403959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:27.403767    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b728339-4a94-4aa4-b75f-c1b16589d9ce-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:28.138578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:28.138549    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"
Nov 03 19:32:28.139023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:28.138579    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq" event={"ID":"2b728339-4a94-4aa4-b75f-c1b16589d9ce","Type":"ContainerDied","Data":"0aff9db4e52cf717f0302325ea18df0416e24730332fb1d87c0a577bbdfb4bbe"}
Nov 03 19:32:28.139023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:28.138637    4944 scope.go:122] "RemoveContainer" containerID="ba9f7a800b031b9926e535c13c132f7e03bf2a527bec062d1bb9e25394e077fa"
Nov 03 19:32:28.338993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:28.338957    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:32:28.385168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:28.385135    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c3e530-openshift-default-7987b465dc-26tcq"]
Nov 03 19:32:29.857947 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:29.857910    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:29.933175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:29.933140    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" path="/var/lib/kubelet/pods/2b728339-4a94-4aa4-b75f-c1b16589d9ce/volumes"
Nov 03 19:32:33.281862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:33.281829    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:33.535033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:33.534956    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:34.171315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:34.171264    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:32:34.260581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:34.260540    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:32:34.357612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:34.357576    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:34.632649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:34.632604    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:34.878820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:34.878751    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:35.224329 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:32:35.224294    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:32:37.084981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:37.084950    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:37.335044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:37.334940    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:38.994343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:38.994305    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:39.172694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:39.172639    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing" containerID="cri-o://572266df77b2eebb0057f5a69776c313250daa5861b6e4719b2ed05d55943d86" gracePeriod=25
Nov 03 19:32:39.261416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:39.261330    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing" containerID="cri-o://ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0" gracePeriod=25
Nov 03 19:32:39.298406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:39.298375    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:43.549496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:43.549462    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:48.074290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:48.074242    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:48.289849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:48.289814    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:49.299222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:49.299187    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:52.701606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:52.701573    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:52.949336 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:52.949299    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:53.201460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:53.201425    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:53.613115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:53.613080    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:53.857725 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:53.857691    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:57.473045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:57.473002    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:32:58.289530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:58.289494    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:32:59.298332 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:32:59.298308    4944 logging.go:55] [core] [Channel #10077 SubChannel #10078]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.208:50052", ServerName: "10.134.0.208:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.208:50052: connect: connection refused"
Nov 03 19:32:59.330485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.330457    4944 generic.go:356] "Generic (PLEG): container finished" podID="77f3a3d0-430d-4591-b943-e2231afb753e" containerID="572266df77b2eebb0057f5a69776c313250daa5861b6e4719b2ed05d55943d86" exitCode=0
Nov 03 19:32:59.330600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.330532    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" event={"ID":"77f3a3d0-430d-4591-b943-e2231afb753e","Type":"ContainerDied","Data":"572266df77b2eebb0057f5a69776c313250daa5861b6e4719b2ed05d55943d86"}
Nov 03 19:32:59.518860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.518834    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:32:59.539760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.539727    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:32:59.642320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.642281    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\") pod \"77f3a3d0-430d-4591-b943-e2231afb753e\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") "
Nov 03 19:32:59.642501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.642371    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\" (UniqueName: \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\") pod \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") "
Nov 03 19:32:59.642578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.642504    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\" (UniqueName: \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\") pod \"77f3a3d0-430d-4591-b943-e2231afb753e\" (UID: \"77f3a3d0-430d-4591-b943-e2231afb753e\") "
Nov 03 19:32:59.642641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.642597    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\") pod \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\" (UID: \"d5a35f35-56da-4a98-85c6-1efb41f2aadc\") "
Nov 03 19:32:59.642700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.642673    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume" pod "77f3a3d0-430d-4591-b943-e2231afb753e" (UID: "77f3a3d0-430d-4591-b943-e2231afb753e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:59.643040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.643016    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77f3a3d0-430d-4591-b943-e2231afb753e-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:59.643148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.643057    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume" pod "d5a35f35-56da-4a98-85c6-1efb41f2aadc" (UID: "d5a35f35-56da-4a98-85c6-1efb41f2aadc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:32:59.644595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.644573    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt" pod "d5a35f35-56da-4a98-85c6-1efb41f2aadc" (UID: "d5a35f35-56da-4a98-85c6-1efb41f2aadc"). InnerVolumeSpecName "kube-api-access-skljt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:59.644681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.644618    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6" pod "77f3a3d0-430d-4591-b943-e2231afb753e" (UID: "77f3a3d0-430d-4591-b943-e2231afb753e"). InnerVolumeSpecName "kube-api-access-4t9g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:32:59.744207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.744181    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skljt\" (UniqueName: \"kubernetes.io/projected/d5a35f35-56da-4a98-85c6-1efb41f2aadc-kube-api-access-skljt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:59.744366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.744210    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t9g6\" (UniqueName: \"kubernetes.io/projected/77f3a3d0-430d-4591-b943-e2231afb753e-kube-api-access-4t9g6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:32:59.744366 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:32:59.744226    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5a35f35-56da-4a98-85c6-1efb41f2aadc-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:00.299228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.299184    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.208:50052\" within 1s: context deadline exceeded"
Nov 03 19:33:00.299654 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:33:00.299233    4944 logging.go:55] [core] [Channel #10077 SubChannel #10078]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.208:50052", ServerName: "10.134.0.208:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.208:50052: operation was canceled"
Nov 03 19:33:00.299654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.299315    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:33:00.337840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.337811    4944 generic.go:356] "Generic (PLEG): container finished" podID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerID="ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0" exitCode=0
Nov 03 19:33:00.338010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.337855    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"
Nov 03 19:33:00.338010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.337888    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" event={"ID":"d5a35f35-56da-4a98-85c6-1efb41f2aadc","Type":"ContainerDied","Data":"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"}
Nov 03 19:33:00.338010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.337948    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w" event={"ID":"d5a35f35-56da-4a98-85c6-1efb41f2aadc","Type":"ContainerDied","Data":"68abb4f1b6fb639c2b1402ba24c1d8a6401d27f928a1da47d031052cb4268b5b"}
Nov 03 19:33:00.338010 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.337977    4944 scope.go:122] "RemoveContainer" containerID="ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"
Nov 03 19:33:00.339555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.339530    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"
Nov 03 19:33:00.339663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.339565    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2" event={"ID":"77f3a3d0-430d-4591-b943-e2231afb753e","Type":"ContainerDied","Data":"955f4eb36756caa9d5e65cd90ae42dad0ac95d641af314decdc6cfa11d1e65aa"}
Nov 03 19:33:00.347588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.347572    4944 scope.go:122] "RemoveContainer" containerID="ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"
Nov 03 19:33:00.347944 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:33:00.347846    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0\": container with ID starting with ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0 not found: ID does not exist" containerID="ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"
Nov 03 19:33:00.347944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.347871    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0"} err="failed to get container status \"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0\": rpc error: code = NotFound desc = could not find container \"ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0\": container with ID starting with ab6c5d76e6d5a49d12de263f6403718e250bdd702888e3ebcbb4417309a5c9f0 not found: ID does not exist"
Nov 03 19:33:00.347944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.347886    4944 scope.go:122] "RemoveContainer" containerID="572266df77b2eebb0057f5a69776c313250daa5861b6e4719b2ed05d55943d86"
Nov 03 19:33:00.491490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.491461    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:33:00.497312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.497286    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-73a216-88dfbd48d-lchw2"]
Nov 03 19:33:00.599316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.599281    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:33:00.601702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:00.601681    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-73a216-7d4dfd467-grx4w"]
Nov 03 19:33:01.396578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:01.396519    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:01.934236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:01.934199    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" path="/var/lib/kubelet/pods/77f3a3d0-430d-4591-b943-e2231afb753e/volumes"
Nov 03 19:33:01.934613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:01.934599    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" path="/var/lib/kubelet/pods/d5a35f35-56da-4a98-85c6-1efb41f2aadc/volumes"
Nov 03 19:33:02.884087 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:02.884050    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:03.222957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.222870    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:03.276809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.276760    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"]
Nov 03 19:33:03.322721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.322692    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-010ed3-xfq9j"]
Nov 03 19:33:03.475090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.475014    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:03.776655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.776625    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:03.933581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:03.933543    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba44825-3049-4613-a789-fe5931454a43" path="/var/lib/kubelet/pods/aba44825-3049-4613-a789-fe5931454a43/volumes"
Nov 03 19:33:05.158498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:05.158466    4944 scope.go:122] "RemoveContainer" containerID="7277b42c8c197a0c389475104e331892789c967b736ce908871af7674636e074"
Nov 03 19:33:05.168402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:05.168377    4944 scope.go:122] "RemoveContainer" containerID="1ea9d35e0f1c47943db5306698e664c2cd6502b09f514a98f7340c615ee22ba0"
Nov 03 19:33:05.228250 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:33:05.228221    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:33:07.139705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:07.139655    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:08.734114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:08.734085    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:08.985109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:08.985021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:09.024618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:09.024575    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"]
Nov 03 19:33:09.070417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:09.070385    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-73a216-bg766"]
Nov 03 19:33:09.233769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:09.233714    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:09.534525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:09.534492    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:09.939978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:09.939898    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6030ac-6021-4b5f-aef5-1dbcfe683381" path="/var/lib/kubelet/pods/0d6030ac-6021-4b5f-aef5-1dbcfe683381/volumes"
Nov 03 19:33:10.227341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:10.227229    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:10.474187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:10.474146    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:10.732106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:10.732067    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:13.299990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:13.299948    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:14.756015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:14.755979    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:15.011873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.007337    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:15.143178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.143140    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"]
Nov 03 19:33:15.189332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.189299    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-9dcd6c-vg2df"]
Nov 03 19:33:15.272159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.272076    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:15.640872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.640766    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:15.889663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.889619    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:15.934543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:15.934471    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b03ce2-5894-42c8-b87c-462330e01902" path="/var/lib/kubelet/pods/34b03ce2-5894-42c8-b87c-462330e01902/volumes"
Nov 03 19:33:16.170190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.170144    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:16.648256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648219    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"]
Nov 03 19:33:16.648758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648743    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0d6030ac-6021-4b5f-aef5-1dbcfe683381" containerName="revoke-keys"
Nov 03 19:33:16.648822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648759    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6030ac-6021-4b5f-aef5-1dbcfe683381" containerName="revoke-keys"
Nov 03 19:33:16.648822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648772    4944 cpu_manager.go:414] "containerMap: removing container" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing"
Nov 03 19:33:16.648822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648777    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing"
Nov 03 19:33:16.648822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648802    4944 cpu_manager.go:414] "containerMap: removing container" podUID="34b03ce2-5894-42c8-b87c-462330e01902" containerName="revoke-keys"
Nov 03 19:33:16.648822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648810    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b03ce2-5894-42c8-b87c-462330e01902" containerName="revoke-keys"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648827    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648832    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648854    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648859    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648949    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a35f35-56da-4a98-85c6-1efb41f2aadc" containerName="payload-pre-processing"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648960    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6030ac-6021-4b5f-aef5-1dbcfe683381" containerName="revoke-keys"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648968    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f3a3d0-430d-4591-b943-e2231afb753e" containerName="payload-processing"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648980    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b03ce2-5894-42c8-b87c-462330e01902" containerName="revoke-keys"
Nov 03 19:33:16.649033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.648986    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b728339-4a94-4aa4-b75f-c1b16589d9ce" containerName="istio-proxy"
Nov 03 19:33:16.665108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.665082    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.715559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.708338    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"]
Nov 03 19:33:16.750325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.750289    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:33:16.750325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.750304    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:33:16.750588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.750437    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:33:16.762199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.762167    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:33:16.817889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.817854    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.818111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.817896    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.818111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.818072    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjcw\" (UniqueName: \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.919893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.919770    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjcw\" (UniqueName: \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.920351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.919945    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.920351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.919977    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.920775 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.920749    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.922413 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.922365    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.977063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.977028    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjcw\" (UniqueName: \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\") pod \"maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:16.978862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:16.978840    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:17.350873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:17.350820    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"]
Nov 03 19:33:17.356686 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:33:17.356659    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387a99aa_5f5b_4aae_9ae0_08c5a9468531.slice/crio-0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d WatchSource:0}: Error finding container 0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d: Status 404 returned error can't find the container with id 0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d
Nov 03 19:33:17.459479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:17.459433    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k" event={"ID":"387a99aa-5f5b-4aae-9ae0-08c5a9468531","Type":"ContainerStarted","Data":"0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d"}
Nov 03 19:33:18.475742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:18.475702    4944 generic.go:356] "Generic (PLEG): container finished" podID="387a99aa-5f5b-4aae-9ae0-08c5a9468531" containerID="2cc7638dde390c94abaf1533a61fef9d13bd30e3c9511b31c8cc068f28efd1fa" exitCode=0
Nov 03 19:33:18.476228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:18.475745    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k" event={"ID":"387a99aa-5f5b-4aae-9ae0-08c5a9468531","Type":"ContainerDied","Data":"2cc7638dde390c94abaf1533a61fef9d13bd30e3c9511b31c8cc068f28efd1fa"}
Nov 03 19:33:19.483172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.483137    4944 generic.go:356] "Generic (PLEG): container finished" podID="a23466c0-a35e-4646-b152-859d188fa066" containerID="966b288281c61d6cb01a21e9b2576cd242ca2ed78c301507b8397879e1663811" exitCode=0
Nov 03 19:33:19.483724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.483212    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerDied","Data":"966b288281c61d6cb01a21e9b2576cd242ca2ed78c301507b8397879e1663811"}
Nov 03 19:33:19.483724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.483709    4944 scope.go:122] "RemoveContainer" containerID="966b288281c61d6cb01a21e9b2576cd242ca2ed78c301507b8397879e1663811"
Nov 03 19:33:19.485042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.485012    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:33:19.485156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.485079    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:33:19.485156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.485118    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:33:19.638831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.638780    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:19.752502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.752459    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\") pod \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") "
Nov 03 19:33:19.753303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.753272    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\") pod \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") "
Nov 03 19:33:19.753431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.753358    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\" (UniqueName: \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\") pod \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\" (UID: \"387a99aa-5f5b-4aae-9ae0-08c5a9468531\") "
Nov 03 19:33:19.753873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.753822    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca" pod "387a99aa-5f5b-4aae-9ae0-08c5a9468531" (UID: "387a99aa-5f5b-4aae-9ae0-08c5a9468531"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:19.756306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.756276    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca" pod "387a99aa-5f5b-4aae-9ae0-08c5a9468531" (UID: "387a99aa-5f5b-4aae-9ae0-08c5a9468531"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:33:19.756416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.756351    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw" pod "387a99aa-5f5b-4aae-9ae0-08c5a9468531" (UID: "387a99aa-5f5b-4aae-9ae0-08c5a9468531"). InnerVolumeSpecName "kube-api-access-8gjcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:33:19.854415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.854374    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gjcw\" (UniqueName: \"kubernetes.io/projected/387a99aa-5f5b-4aae-9ae0-08c5a9468531-kube-api-access-8gjcw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:19.854415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.854414    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:19.854678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.854429    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/387a99aa-5f5b-4aae-9ae0-08c5a9468531-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:19.963109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:19.963068    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:20.494832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.494777    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerStarted","Data":"1572c36546352c277c7ac07831f1647850c3b63dff336274d39558806ccba94b"}
Nov 03 19:33:20.496386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.496363    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"
Nov 03 19:33:20.496501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.496384    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k" event={"ID":"387a99aa-5f5b-4aae-9ae0-08c5a9468531","Type":"ContainerDied","Data":"0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d"}
Nov 03 19:33:20.496501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.496423    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b14a326afaeb0aaf186600e95cdb21b63fdef42ec430de7e7186c863944820d"
Nov 03 19:33:20.634063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.634024    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:33:20.635520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.635492    4944 patch_prober.go:29] interesting pod/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 19:33:20.635673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:20.635543    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 19:33:21.634502 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:21.634464    4944 patch_prober.go:29] interesting pod/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 19:33:21.635002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:21.634527    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 19:33:21.792720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:21.792679    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:21.973095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:21.973008    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:33:22.043141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.043111    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:22.286928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.286896    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:22.584357 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.584279    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"]
Nov 03 19:33:22.584767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.584753    4944 cpu_manager.go:414] "containerMap: removing container" podUID="387a99aa-5f5b-4aae-9ae0-08c5a9468531" containerName="revoke-keys"
Nov 03 19:33:22.584841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.584769    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387a99aa-5f5b-4aae-9ae0-08c5a9468531" containerName="revoke-keys"
Nov 03 19:33:22.584884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.584856    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="387a99aa-5f5b-4aae-9ae0-08c5a9468531" containerName="revoke-keys"
Nov 03 19:33:22.587268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.587248    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.634425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.634383    4944 patch_prober.go:29] interesting pod/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 19:33:22.634609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.634441    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 19:33:22.682742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682709    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:33:22.682742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682724    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:33:22.682742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682774    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:33:22.683067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682836    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkvph\" (UniqueName: \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.683067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682931    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:33:22.683067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.682949    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.683067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.683035    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.690450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.688689    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"]
Nov 03 19:33:22.784397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.784356    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.784577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.784471    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkvph\" (UniqueName: \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.784577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.784515    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.785031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.785007    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.786830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.786803    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.841980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.841904    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkvph\" (UniqueName: \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:22.899466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:22.899433    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:23.302287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:23.302247    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"]
Nov 03 19:33:23.333321 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:33:23.333286    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00440078_5bd3_41f6_b15f_06010dee295a.slice/crio-9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7 WatchSource:0}: Error finding container 9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7: Status 404 returned error can't find the container with id 9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7
Nov 03 19:33:23.539164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:23.539091    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g" event={"ID":"00440078-5bd3-41f6-b15f-06010dee295a","Type":"ContainerStarted","Data":"9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7"}
Nov 03 19:33:23.634275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:23.634211    4944 patch_prober.go:29] interesting pod/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 19:33:23.634468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:23.634275    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 19:33:24.546758 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:24.546721    4944 generic.go:356] "Generic (PLEG): container finished" podID="00440078-5bd3-41f6-b15f-06010dee295a" containerID="27137cfc288c6074185a97e73835029112cdaf4ff7d5a67b8132af3266e9df51" exitCode=0
Nov 03 19:33:24.547183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:24.546829    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g" event={"ID":"00440078-5bd3-41f6-b15f-06010dee295a","Type":"ContainerDied","Data":"27137cfc288c6074185a97e73835029112cdaf4ff7d5a67b8132af3266e9df51"}
Nov 03 19:33:24.634391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:24.634301    4944 patch_prober.go:29] interesting pod/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body=
Nov 03 19:33:24.634391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:24.634363    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused"
Nov 03 19:33:24.733074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:24.733042    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:25.666688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.666650    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:33:25.668665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.668639    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:33:25.707178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.707155    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:25.834204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.834124    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\") pod \"00440078-5bd3-41f6-b15f-06010dee295a\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") "
Nov 03 19:33:25.834204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.834168    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\" (UniqueName: \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\") pod \"00440078-5bd3-41f6-b15f-06010dee295a\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") "
Nov 03 19:33:25.834204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.834195    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\") pod \"00440078-5bd3-41f6-b15f-06010dee295a\" (UID: \"00440078-5bd3-41f6-b15f-06010dee295a\") "
Nov 03 19:33:25.834684 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.834562    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca" pod "00440078-5bd3-41f6-b15f-06010dee295a" (UID: "00440078-5bd3-41f6-b15f-06010dee295a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:25.835180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.834911    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:25.836774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.836750    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca" pod "00440078-5bd3-41f6-b15f-06010dee295a" (UID: "00440078-5bd3-41f6-b15f-06010dee295a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:33:25.836774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.836755    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph" pod "00440078-5bd3-41f6-b15f-06010dee295a" (UID: "00440078-5bd3-41f6-b15f-06010dee295a"). InnerVolumeSpecName "kube-api-access-vkvph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:33:25.936121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.936092    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkvph\" (UniqueName: \"kubernetes.io/projected/00440078-5bd3-41f6-b15f-06010dee295a-kube-api-access-vkvph\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:25.936121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:25.936118    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00440078-5bd3-41f6-b15f-06010dee295a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:26.561175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:26.561144    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"
Nov 03 19:33:26.561389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:26.561180    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g" event={"ID":"00440078-5bd3-41f6-b15f-06010dee295a","Type":"ContainerDied","Data":"9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7"}
Nov 03 19:33:26.561389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:26.561241    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e59288ea3659047329f5eb8ab4ee074991ba6fb6d753b96b70f722ae20cdda7"
Nov 03 19:33:28.300718 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:28.300671    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:33:28.401291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:28.401257    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:33:28.757446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:28.757414    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:29.012186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:29.012090    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:29.637370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:29.637330    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:31.046224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:31.046186    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"]
Nov 03 19:33:31.092225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:31.092191    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc45999-vmvxz"]
Nov 03 19:33:31.933288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:31.933255    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dffdf7-4e77-4d89-8d16-c0d141d96890" path="/var/lib/kubelet/pods/64dffdf7-4e77-4d89-8d16-c0d141d96890/volumes"
Nov 03 19:33:33.301477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.301422    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" podUID="29712e8d-fac3-4466-99d0-6f38903ce8fb" containerName="payload-processing" containerID="cri-o://5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9" gracePeriod=25
Nov 03 19:33:33.563620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.563561    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:33:33.573466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.573431    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing" containerID="cri-o://f6ccc253e394257039bd66ff52e31de3ca624c8007553ee7f69227089e4cafd9" gracePeriod=25
Nov 03 19:33:33.608249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608219    4944 generic.go:356] "Generic (PLEG): container finished" podID="29712e8d-fac3-4466-99d0-6f38903ce8fb" containerID="5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9" exitCode=0
Nov 03 19:33:33.608421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608262    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" event={"ID":"29712e8d-fac3-4466-99d0-6f38903ce8fb","Type":"ContainerDied","Data":"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"}
Nov 03 19:33:33.608421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608276    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"
Nov 03 19:33:33.608421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608325    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c" event={"ID":"29712e8d-fac3-4466-99d0-6f38903ce8fb","Type":"ContainerDied","Data":"789a101a72e3367787626b0e09a2c40c4f764df8a8b4201e284fedeb1c25dcb5"}
Nov 03 19:33:33.608421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608348    4944 scope.go:122] "RemoveContainer" containerID="5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"
Nov 03 19:33:33.608976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608951    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\") pod \"29712e8d-fac3-4466-99d0-6f38903ce8fb\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") "
Nov 03 19:33:33.609076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.608996    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\" (UniqueName: \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\") pod \"29712e8d-fac3-4466-99d0-6f38903ce8fb\" (UID: \"29712e8d-fac3-4466-99d0-6f38903ce8fb\") "
Nov 03 19:33:33.609356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.609321    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume" pod "29712e8d-fac3-4466-99d0-6f38903ce8fb" (UID: "29712e8d-fac3-4466-99d0-6f38903ce8fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:33.609464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.609415    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29712e8d-fac3-4466-99d0-6f38903ce8fb-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:33.611330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.611307    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9" pod "29712e8d-fac3-4466-99d0-6f38903ce8fb" (UID: "29712e8d-fac3-4466-99d0-6f38903ce8fb"). InnerVolumeSpecName "kube-api-access-56kc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:33:33.614309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.614288    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:33.624559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.624537    4944 scope.go:122] "RemoveContainer" containerID="5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"
Nov 03 19:33:33.624849 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:33:33.624828    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9\": container with ID starting with 5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9 not found: ID does not exist" containerID="5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"
Nov 03 19:33:33.624909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.624856    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9"} err="failed to get container status \"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9\": rpc error: code = NotFound desc = could not find container \"5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9\": container with ID starting with 5c8ff7c2691ced5c3e1948faf0e616d072a13ab6405f05188ec3cb1f77148bb9 not found: ID does not exist"
Nov 03 19:33:33.710071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:33.710039    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kc9\" (UniqueName: \"kubernetes.io/projected/29712e8d-fac3-4466-99d0-6f38903ce8fb-kube-api-access-56kc9\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:34.021531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:34.021492    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:33:34.024820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:34.024781    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22ed3b-554c74b886-7zz7c"]
Nov 03 19:33:34.690217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:34.690170    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:34.938084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:34.938037    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:35.224352 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:33:35.224314    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:33:35.461218 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:35.461183    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:33:35.933299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:35.933265    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29712e8d-fac3-4466-99d0-6f38903ce8fb" path="/var/lib/kubelet/pods/29712e8d-fac3-4466-99d0-6f38903ce8fb/volumes"
Nov 03 19:33:39.652516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:39.652479    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:40.716698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:40.716660    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:43.012857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.012818    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:33:43.013309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013296    4944 cpu_manager.go:414] "containerMap: removing container" podUID="29712e8d-fac3-4466-99d0-6f38903ce8fb" containerName="payload-processing"
Nov 03 19:33:43.013367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013310    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29712e8d-fac3-4466-99d0-6f38903ce8fb" containerName="payload-processing"
Nov 03 19:33:43.013367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013324    4944 cpu_manager.go:414] "containerMap: removing container" podUID="00440078-5bd3-41f6-b15f-06010dee295a" containerName="revoke-keys"
Nov 03 19:33:43.013367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013329    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00440078-5bd3-41f6-b15f-06010dee295a" containerName="revoke-keys"
Nov 03 19:33:43.013467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013401    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="29712e8d-fac3-4466-99d0-6f38903ce8fb" containerName="payload-processing"
Nov 03 19:33:43.013467 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.013412    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="00440078-5bd3-41f6-b15f-06010dee295a" containerName="revoke-keys"
Nov 03 19:33:43.018134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.018115    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.058769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.058732    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"]
Nov 03 19:33:43.061003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.060979    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db361c36-qlpcg"]
Nov 03 19:33:43.098819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.098774    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fcb646\"/\"kube-root-ca.crt\""
Nov 03 19:33:43.100305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100274    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.100464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100349    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.100464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100438    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.100595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100490    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chl7j\" (UniqueName: \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.100650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100610    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.100650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.100641    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.104145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.104121    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fcb646\"/\"default-dockercfg-xdgq7\""
Nov 03 19:33:43.104343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.104172    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fcb646\"/\"openshift-service-ca.crt\""
Nov 03 19:33:43.104343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.104213    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fcb646\"/\"test-model-fcb646-kserve-self-signed-certs\""
Nov 03 19:33:43.108493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.108470    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:33:43.201723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201693    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.201923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201734    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.201923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201769    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.201923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201822    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chl7j\" (UniqueName: \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.201923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201883    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.201923 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.201899    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.202239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.202162    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.202300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.202275    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.202494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.202470    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.204160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.204136    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.204637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.204612    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.246028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.245999    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:43.258918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.258894    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chl7j\" (UniqueName: \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\") pod \"test-model-fcb646-kserve-5bd99c749-48vvk\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") " pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.330405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.330319    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:43.701309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.701222    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:33:43.707139 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:33:43.707108    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e23bda4_c7a2_4f96_8410_b59b46b89069.slice/crio-75384942c60f10bd4983db7217015a279fdf1d5834c805aa893d7d2aa7f238cb WatchSource:0}: Error finding container 75384942c60f10bd4983db7217015a279fdf1d5834c805aa893d7d2aa7f238cb: Status 404 returned error can't find the container with id 75384942c60f10bd4983db7217015a279fdf1d5834c805aa893d7d2aa7f238cb
Nov 03 19:33:43.934604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:43.934569    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34c2d52-6f21-4157-944d-9a95f76433a5" path="/var/lib/kubelet/pods/f34c2d52-6f21-4157-944d-9a95f76433a5/volumes"
Nov 03 19:33:44.466232 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:44.466198    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:44.689266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:44.689228    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" event={"ID":"9e23bda4-c7a2-4f96-8410-b59b46b89069","Type":"ContainerStarted","Data":"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"}
Nov 03 19:33:44.689438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:44.689276    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" event={"ID":"9e23bda4-c7a2-4f96-8410-b59b46b89069","Type":"ContainerStarted","Data":"75384942c60f10bd4983db7217015a279fdf1d5834c805aa893d7d2aa7f238cb"}
Nov 03 19:33:44.798339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:44.798290    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" podStartSLOduration=2.79827459 podStartE2EDuration="2.79827459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:33:44.798135795 +0000 UTC m=+2923.075489378" watchObservedRunningTime="2026-11-03 19:33:44.79827459 +0000 UTC m=+2923.075628175"
Nov 03 19:33:45.330528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:45.330498    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:45.460724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:45.460676    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:33:48.995468 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:48.995371    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:53.084878 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.084837    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:53.330760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.330727    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:53.342959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.342898    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:53.741475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.741440    4944 generic.go:356] "Generic (PLEG): container finished" podID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerID="f6ccc253e394257039bd66ff52e31de3ca624c8007553ee7f69227089e4cafd9" exitCode=0
Nov 03 19:33:53.741640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.741513    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" event={"ID":"313f1526-a0cd-4c24-9d5b-d5918a4c379c","Type":"ContainerDied","Data":"f6ccc253e394257039bd66ff52e31de3ca624c8007553ee7f69227089e4cafd9"}
Nov 03 19:33:53.761286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.761247    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:33:53.838060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.838037    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:33:53.919286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.919254    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\" (UniqueName: \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\") pod \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") "
Nov 03 19:33:53.919478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.919413    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\") pod \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\" (UID: \"313f1526-a0cd-4c24-9d5b-d5918a4c379c\") "
Nov 03 19:33:53.919753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.919732    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume" pod "313f1526-a0cd-4c24-9d5b-d5918a4c379c" (UID: "313f1526-a0cd-4c24-9d5b-d5918a4c379c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:33:53.919877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.919862    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313f1526-a0cd-4c24-9d5b-d5918a4c379c-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:53.921314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:53.921289    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g" pod "313f1526-a0cd-4c24-9d5b-d5918a4c379c" (UID: "313f1526-a0cd-4c24-9d5b-d5918a4c379c"). InnerVolumeSpecName "kube-api-access-2hd6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:33:54.020734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.020689    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hd6g\" (UniqueName: \"kubernetes.io/projected/313f1526-a0cd-4c24-9d5b-d5918a4c379c-kube-api-access-2hd6g\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:33:54.536920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.536888    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:54.748803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.748770    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"
Nov 03 19:33:54.748996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.748806    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d" event={"ID":"313f1526-a0cd-4c24-9d5b-d5918a4c379c","Type":"ContainerDied","Data":"34403f94f505a7f9ecebd355d3dcd1b3f669847b903d4dc1393967eacf93b0eb"}
Nov 03 19:33:54.748996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.748898    4944 scope.go:122] "RemoveContainer" containerID="f6ccc253e394257039bd66ff52e31de3ca624c8007553ee7f69227089e4cafd9"
Nov 03 19:33:54.808022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.807946    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:54.924271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.924239    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:33:54.942291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.942260    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-22ed3b-65869b8567-rdv9d"]
Nov 03 19:33:54.970513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.970480    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"]
Nov 03 19:33:54.986563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:54.986533    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-22ed3b-lj28k"]
Nov 03 19:33:55.148724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:55.148640    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:55.447981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:55.447894    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:33:55.936481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:55.936439    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" path="/var/lib/kubelet/pods/313f1526-a0cd-4c24-9d5b-d5918a4c379c/volumes"
Nov 03 19:33:55.937037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:55.936953    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387a99aa-5f5b-4aae-9ae0-08c5a9468531" path="/var/lib/kubelet/pods/387a99aa-5f5b-4aae-9ae0-08c5a9468531/volumes"
Nov 03 19:33:59.620521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:33:59.620484    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:02.321140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:02.321103    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:03.267703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.267674    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"]
Nov 03 19:34:03.268146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.268133    4944 cpu_manager.go:414] "containerMap: removing container" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing"
Nov 03 19:34:03.268208 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.268147    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing"
Nov 03 19:34:03.268249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.268241    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="313f1526-a0cd-4c24-9d5b-d5918a4c379c" containerName="payload-pre-processing"
Nov 03 19:34:03.271497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.271481    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.326354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.326323    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"]
Nov 03 19:34:03.368985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.368951    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:34:03.369165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.368942    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:34:03.369165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.368955    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:34:03.369165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.369031    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:34:03.417742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.417707    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.417912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.417802    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.417912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.417837    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5q4\" (UniqueName: \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.519533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.519435    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.519533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.519479    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5q4\" (UniqueName: \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.519779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.519576    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.520233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.520214    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.521927 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.521901    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.577224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.577194    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5q4\" (UniqueName: \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\") pod \"maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.584067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.584045    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:03.972073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:03.972044    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"]
Nov 03 19:34:03.972360 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:34:03.972334    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fb5ae9_bbfe_4d31_8383_fbea5d6440dd.slice/crio-333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36 WatchSource:0}: Error finding container 333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36: Status 404 returned error can't find the container with id 333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36
Nov 03 19:34:04.814535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:04.814504    4944 generic.go:356] "Generic (PLEG): container finished" podID="22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" containerID="5314e44d2a36331b81658c018f089dbda4b85fa0bba038bcc83cebeb7a77f1b5" exitCode=0
Nov 03 19:34:04.814889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:04.814590    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc" event={"ID":"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd","Type":"ContainerDied","Data":"5314e44d2a36331b81658c018f089dbda4b85fa0bba038bcc83cebeb7a77f1b5"}
Nov 03 19:34:04.814889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:04.814680    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc" event={"ID":"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd","Type":"ContainerStarted","Data":"333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36"}
Nov 03 19:34:04.884582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:04.884551    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.223193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.223121    4944 scope.go:122] "RemoveContainer" containerID="fd5ba0c5a7177fa098335d42992bf45d6591f07e8c598b4060a6fa480a6e2036"
Nov 03 19:34:05.223861 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:05.223832    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:34:05.377179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.377146    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.654190 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.654149    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:05.889535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.889500    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"]
Nov 03 19:34:05.934687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.934660    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-826a68-6d57g"]
Nov 03 19:34:05.957639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.957611    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:05.970917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:05.970893    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:06.044511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.044487    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\") pod \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") "
Nov 03 19:34:06.044675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.044521    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\" (UniqueName: \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\") pod \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") "
Nov 03 19:34:06.044675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.044575    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\") pod \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\" (UID: \"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd\") "
Nov 03 19:34:06.044921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.044892    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca" pod "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" (UID: "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:06.045196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.045176    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:06.046678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.046657    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca" pod "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" (UID: "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:06.046845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.046827    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4" pod "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" (UID: "22fb5ae9-bbfe-4d31-8383-fbea5d6440dd"). InnerVolumeSpecName "kube-api-access-hw5q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:06.146013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.145981    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5q4\" (UniqueName: \"kubernetes.io/projected/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-kube-api-access-hw5q4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:06.146013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.146008    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:06.378187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.378152    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:06.826765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.826725    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc" event={"ID":"22fb5ae9-bbfe-4d31-8383-fbea5d6440dd","Type":"ContainerDied","Data":"333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36"}
Nov 03 19:34:06.826765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.826757    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"
Nov 03 19:34:06.826990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:06.826816    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="333b677eacbff08bf05453b383ae4089273b11dfd0fabf3fa8c1b1bdc6834d36"
Nov 03 19:34:07.933991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:07.933958    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00440078-5bd3-41f6-b15f-06010dee295a" path="/var/lib/kubelet/pods/00440078-5bd3-41f6-b15f-06010dee295a/volumes"
Nov 03 19:34:07.964627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:07.964587    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:34:08.009550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:08.009518    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 19:34:08.170800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:08.170747    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:10.630628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:10.630597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:12.820498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:12.820466    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:13.094174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.094087    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:13.360694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.360608    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:13.729043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.728962    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:13.818191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.818157    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"]
Nov 03 19:34:13.818624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.818611    4944 cpu_manager.go:414] "containerMap: removing container" podUID="22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" containerName="revoke-keys"
Nov 03 19:34:13.818670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.818625    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" containerName="revoke-keys"
Nov 03 19:34:13.818716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.818707    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" containerName="revoke-keys"
Nov 03 19:34:13.822211 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.822192    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:13.876525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.876485    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"]
Nov 03 19:34:13.919309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.919283    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:34:13.919309 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.919307    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:34:13.927269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.927245    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:13.927394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.927365    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:13.927657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.927635    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkr6h\" (UniqueName: \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:13.930176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.930157    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:34:13.931202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:13.931164    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:34:14.028597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.028571    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.028773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.028718    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkr6h\" (UniqueName: \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.028773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.028748    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.029280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.029246    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.031116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.031097    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.101453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.101416    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkr6h\" (UniqueName: \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\") pod \"maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.135819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.135775    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:14.514194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.514159    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"]
Nov 03 19:34:14.880402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.880364    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r" event={"ID":"cfca82d4-94c6-464f-83c6-d171ccf92892","Type":"ContainerStarted","Data":"8650918dae5c53fced0ca58ba4d7a13b34222448bceb87c88d9e9163b302538d"}
Nov 03 19:34:14.880925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.880467    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r" event={"ID":"cfca82d4-94c6-464f-83c6-d171ccf92892","Type":"ContainerStarted","Data":"0f10b6d60fbe0ac372745c4aad544d036ece15049a5a2805dc1bf9b03125331d"}
Nov 03 19:34:14.919829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.919777    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"]
Nov 03 19:34:14.923614 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.923597    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:14.937508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.937486    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:14.937602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.937571    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l76p\" (UniqueName: \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:14.937602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.937594    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:14.995008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:14.994933    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"]
Nov 03 19:34:15.022302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.022252    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r" podStartSLOduration=2.022238256 podStartE2EDuration="2.022238256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:15.020721646 +0000 UTC m=+2953.298075223" watchObservedRunningTime="2026-11-03 19:34:15.022238256 +0000 UTC m=+2953.299591838"
Nov 03 19:34:15.038972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.038942    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.039146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.039068    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l76p\" (UniqueName: \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.039146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.039111    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.040001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.039976    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.041964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.041941    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.098138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.098070    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l76p\" (UniqueName: \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\") pod \"maas-api-revoke-sub-997f7d6d-pk8pt\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.236856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.236822    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:15.608936 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.608901    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"]
Nov 03 19:34:15.616666 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:34:15.616638    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5e1be5_fd9e_4280_8790_076ff15b0c05.slice/crio-36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f WatchSource:0}: Error finding container 36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f: Status 404 returned error can't find the container with id 36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f
Nov 03 19:34:15.837802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.837755    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:15.887550 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.887514    4944 generic.go:356] "Generic (PLEG): container finished" podID="cfca82d4-94c6-464f-83c6-d171ccf92892" containerID="8650918dae5c53fced0ca58ba4d7a13b34222448bceb87c88d9e9163b302538d" exitCode=0
Nov 03 19:34:15.887965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.887596    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r" event={"ID":"cfca82d4-94c6-464f-83c6-d171ccf92892","Type":"ContainerDied","Data":"8650918dae5c53fced0ca58ba4d7a13b34222448bceb87c88d9e9163b302538d"}
Nov 03 19:34:15.889017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:15.888995    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt" event={"ID":"5f5e1be5-fd9e-4280-8790-076ff15b0c05","Type":"ContainerStarted","Data":"36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f"}
Nov 03 19:34:16.896477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:16.896445    4944 generic.go:356] "Generic (PLEG): container finished" podID="5f5e1be5-fd9e-4280-8790-076ff15b0c05" containerID="f23c63ac2ce3fd22551bf185ad6a627a61a319edb25c4b8f29a9f84a39a382ef" exitCode=0
Nov 03 19:34:16.896932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:16.896530    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt" event={"ID":"5f5e1be5-fd9e-4280-8790-076ff15b0c05","Type":"ContainerDied","Data":"f23c63ac2ce3fd22551bf185ad6a627a61a319edb25c4b8f29a9f84a39a382ef"}
Nov 03 19:34:17.024382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.024354    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:17.060251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.060225    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\") pod \"cfca82d4-94c6-464f-83c6-d171ccf92892\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") "
Nov 03 19:34:17.060373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.060310    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\") pod \"cfca82d4-94c6-464f-83c6-d171ccf92892\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") "
Nov 03 19:34:17.060453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.060436    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\" (UniqueName: \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\") pod \"cfca82d4-94c6-464f-83c6-d171ccf92892\" (UID: \"cfca82d4-94c6-464f-83c6-d171ccf92892\") "
Nov 03 19:34:17.060717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.060687    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca" pod "cfca82d4-94c6-464f-83c6-d171ccf92892" (UID: "cfca82d4-94c6-464f-83c6-d171ccf92892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:17.060879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.060833    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:17.062226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.062200    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca" pod "cfca82d4-94c6-464f-83c6-d171ccf92892" (UID: "cfca82d4-94c6-464f-83c6-d171ccf92892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:17.062422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.062404    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h" pod "cfca82d4-94c6-464f-83c6-d171ccf92892" (UID: "cfca82d4-94c6-464f-83c6-d171ccf92892"). InnerVolumeSpecName "kube-api-access-gkr6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:17.161494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.161425    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkr6h\" (UniqueName: \"kubernetes.io/projected/cfca82d4-94c6-464f-83c6-d171ccf92892-kube-api-access-gkr6h\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:17.161494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.161457    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfca82d4-94c6-464f-83c6-d171ccf92892-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:17.904229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.904189    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r" event={"ID":"cfca82d4-94c6-464f-83c6-d171ccf92892","Type":"ContainerDied","Data":"0f10b6d60fbe0ac372745c4aad544d036ece15049a5a2805dc1bf9b03125331d"}
Nov 03 19:34:17.904701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.904249    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f10b6d60fbe0ac372745c4aad544d036ece15049a5a2805dc1bf9b03125331d"
Nov 03 19:34:17.904701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:17.904206    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"
Nov 03 19:34:18.034953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.034923    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:18.071175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.071147    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\") pod \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") "
Nov 03 19:34:18.071302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.071192    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\") pod \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") "
Nov 03 19:34:18.071302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.071259    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\" (UniqueName: \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\") pod \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\" (UID: \"5f5e1be5-fd9e-4280-8790-076ff15b0c05\") "
Nov 03 19:34:18.071504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.071479    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca" pod "5f5e1be5-fd9e-4280-8790-076ff15b0c05" (UID: "5f5e1be5-fd9e-4280-8790-076ff15b0c05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:18.073315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.073289    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca" pod "5f5e1be5-fd9e-4280-8790-076ff15b0c05" (UID: "5f5e1be5-fd9e-4280-8790-076ff15b0c05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:18.073417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.073368    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p" pod "5f5e1be5-fd9e-4280-8790-076ff15b0c05" (UID: "5f5e1be5-fd9e-4280-8790-076ff15b0c05"). InnerVolumeSpecName "kube-api-access-9l76p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:18.173113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.173030    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:18.173113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.173070    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5e1be5-fd9e-4280-8790-076ff15b0c05-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:18.173113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.173087    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l76p\" (UniqueName: \"kubernetes.io/projected/5f5e1be5-fd9e-4280-8790-076ff15b0c05-kube-api-access-9l76p\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:18.631279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.631244    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 19:34:18.715575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.715539    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:18.913395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.913299    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt" event={"ID":"5f5e1be5-fd9e-4280-8790-076ff15b0c05","Type":"ContainerDied","Data":"36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f"}
Nov 03 19:34:18.913395 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.913369    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36c6a568425a2ec8c8d7ca699474784ef8bb129a92dd133a4da043649926083f"
Nov 03 19:34:18.913969 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:18.913432    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"
Nov 03 19:34:19.084756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:19.084716    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:19.329836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:19.329781    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:21.035879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:21.035839    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:22.479818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.479773    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:34:22.480291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480275    4944 cpu_manager.go:414] "containerMap: removing container" podUID="cfca82d4-94c6-464f-83c6-d171ccf92892" containerName="revoke-keys"
Nov 03 19:34:22.480338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480295    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfca82d4-94c6-464f-83c6-d171ccf92892" containerName="revoke-keys"
Nov 03 19:34:22.480338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480310    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5f5e1be5-fd9e-4280-8790-076ff15b0c05" containerName="revoke-keys"
Nov 03 19:34:22.480338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480316    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5e1be5-fd9e-4280-8790-076ff15b0c05" containerName="revoke-keys"
Nov 03 19:34:22.480434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480393    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfca82d4-94c6-464f-83c6-d171ccf92892" containerName="revoke-keys"
Nov 03 19:34:22.480434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.480404    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5e1be5-fd9e-4280-8790-076ff15b0c05" containerName="revoke-keys"
Nov 03 19:34:22.483877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.483856    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.546117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.546084    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:34:22.582028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.582000    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24cdad\"/\"test-model-24cdad-kserve-self-signed-certs\""
Nov 03 19:34:22.582152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.582000    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24cdad\"/\"kube-root-ca.crt\""
Nov 03 19:34:22.582152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.582000    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-24cdad\"/\"default-dockercfg-gpnj5\""
Nov 03 19:34:22.582152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.582067    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-24cdad\"/\"openshift-service-ca.crt\""
Nov 03 19:34:22.617240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617192    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9g9d\" (UniqueName: \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.617412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617257    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.617412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617299    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.617412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617336    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.617615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617416    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.617615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.617447    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.670565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.670522    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:22.718673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718641    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.718673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718675    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.718930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718739    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9g9d\" (UniqueName: \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.718930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718774    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.718930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718826    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.718930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.718849    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.719156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.719129    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.719231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.719209    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.719275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.719254    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.721022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.721000    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.721273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.721258    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.776113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.776082    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9g9d\" (UniqueName: \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\") pod \"test-model-24cdad-kserve-78b7bbd56c-xzwlv\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") " pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.796100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.796073    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:22.985031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:22.984995    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" containerID="cri-o://df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8" gracePeriod=45
Nov 03 19:34:23.029213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.029137    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main" containerID="cri-o://1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03" gracePeriod=45
Nov 03 19:34:23.171890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.167208    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.99:8000/ready\": dial tcp 10.134.0.99:8000: connect: connection refused"
Nov 03 19:34:23.172064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.171942    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:34:23.180992 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:34:23.180965    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ca30fb7_2245_4353_b16c_c8cbe5389e7f.slice/crio-86a2dca3b0ca9b64aff10337f83843cbab61800d717c49ffb8be3e260f53f153 WatchSource:0}: Error finding container 86a2dca3b0ca9b64aff10337f83843cbab61800d717c49ffb8be3e260f53f153: Status 404 returned error can't find the container with id 86a2dca3b0ca9b64aff10337f83843cbab61800d717c49ffb8be3e260f53f153
Nov 03 19:34:23.278095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.278071    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:34:23.309914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.309891    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:34:23.324081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324057    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324096    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324135    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324170    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324219    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324309    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.324375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324334    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\" (UniqueName: \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\") pod \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\" (UID: \"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d\") "
Nov 03 19:34:23.324545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324410    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.324545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324529    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.324811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324775    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.324912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324824    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.324912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.324839    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.326464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.326438    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.326568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.326512    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:23.326568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.326529    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv" pod "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" (UID: "8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d"). InnerVolumeSpecName "kube-api-access-qt6lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:23.425386 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425345    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.425585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425428    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.425585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425466    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\" (UniqueName: \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.425585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425537    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.425769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425590    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.425769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425621    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.425769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.425724    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\") pod \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\" (UID: \"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8\") "
Nov 03 19:34:23.426063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426028    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.426140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426086    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.426290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426270    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.426351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426296    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.426351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426311    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.426351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426324    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.426351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426339    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt6lv\" (UniqueName: \"kubernetes.io/projected/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d-kube-api-access-qt6lv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.426572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.426354    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.428229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.428198    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "kube-api-access-4brdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:23.428477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.428446    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:23.428576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.428545    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm" pod "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" (UID: "99011f11-bdc6-42b1-b33c-dff9aa5ed8e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:23.527248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.527211    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.527248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.527248    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.527714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.527263    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4brdf\" (UniqueName: \"kubernetes.io/projected/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8-kube-api-access-4brdf\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:23.632615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.632511    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing" containerID="cri-o://dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8" gracePeriod=25
Nov 03 19:34:23.949200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.949172    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" event={"ID":"2ca30fb7-2245-4353-b16c-c8cbe5389e7f","Type":"ContainerStarted","Data":"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"}
Nov 03 19:34:23.949374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.949217    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" event={"ID":"2ca30fb7-2245-4353-b16c-c8cbe5389e7f","Type":"ContainerStarted","Data":"86a2dca3b0ca9b64aff10337f83843cbab61800d717c49ffb8be3e260f53f153"}
Nov 03 19:34:23.950385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.950358    4944 generic.go:356] "Generic (PLEG): container finished" podID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerID="1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03" exitCode=0
Nov 03 19:34:23.950483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.950408    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"
Nov 03 19:34:23.950483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.950432    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" event={"ID":"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8","Type":"ContainerDied","Data":"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"}
Nov 03 19:34:23.950483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.950469    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr" event={"ID":"99011f11-bdc6-42b1-b33c-dff9aa5ed8e8","Type":"ContainerDied","Data":"09cf453243b011983599f827d34882c4c03b9ca5a659d0b1d4c4e140430b46c8"}
Nov 03 19:34:23.950644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.950489    4944 scope.go:122] "RemoveContainer" containerID="1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"
Nov 03 19:34:23.953307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.952137    4944 generic.go:356] "Generic (PLEG): container finished" podID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerID="df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8" exitCode=0
Nov 03 19:34:23.953307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.952195    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"
Nov 03 19:34:23.953307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.952208    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" event={"ID":"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d","Type":"ContainerDied","Data":"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"}
Nov 03 19:34:23.953307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.952244    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv" event={"ID":"8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d","Type":"ContainerDied","Data":"3895464f72661e802d5d427531d4b6481a146a38ceece69a7daeec2af9b0324b"}
Nov 03 19:34:23.960692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.960672    4944 scope.go:122] "RemoveContainer" containerID="1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"
Nov 03 19:34:23.960986 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:23.960965    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03\": container with ID starting with 1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03 not found: ID does not exist" containerID="1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"
Nov 03 19:34:23.961046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.960992    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03"} err="failed to get container status \"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03\": rpc error: code = NotFound desc = could not find container \"1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03\": container with ID starting with 1c37b8cf3d02e61461347ae7fd83247de34b0b1bdb852ca0ebe4c36e5d331b03 not found: ID does not exist"
Nov 03 19:34:23.961046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.961011    4944 scope.go:122] "RemoveContainer" containerID="df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"
Nov 03 19:34:23.970638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.970616    4944 scope.go:122] "RemoveContainer" containerID="df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"
Nov 03 19:34:23.970947 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:23.970920    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8\": container with ID starting with df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8 not found: ID does not exist" containerID="df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"
Nov 03 19:34:23.971040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:23.970949    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8"} err="failed to get container status \"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8\": rpc error: code = NotFound desc = could not find container \"df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8\": container with ID starting with df3eb5a0dce00089404cd096e51b054f0153182ce8b75328e272252cfff751d8 not found: ID does not exist"
Nov 03 19:34:24.061908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.061856    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" podStartSLOduration=2.061842647 podStartE2EDuration="2.061842647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:34:24.059547651 +0000 UTC m=+2962.336901235" watchObservedRunningTime="2026-11-03 19:34:24.061842647 +0000 UTC m=+2962.339196278"
Nov 03 19:34:24.164555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.164520    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:34:24.169992 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.169962    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/distinct-w6-eb6ef8-kserve-6b49f75dc5-h2flv"]
Nov 03 19:34:24.269943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.269909    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 19:34:24.273966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.273942    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-eb6ef8/simulator-w6-eb6ef8-kserve-5c758794b9-rkqnr"]
Nov 03 19:34:24.796899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.796868    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:24.997362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:24.997330    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 19:34:25.094517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.094437    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"]
Nov 03 19:34:25.094932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.094918    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main"
Nov 03 19:34:25.094993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.094932    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main"
Nov 03 19:34:25.094993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.094944    4944 cpu_manager.go:414] "containerMap: removing container" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main"
Nov 03 19:34:25.094993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.094950    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main"
Nov 03 19:34:25.095100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.095014    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" containerName="main"
Nov 03 19:34:25.095100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.095023    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" containerName="main"
Nov 03 19:34:25.099808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.099773    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.103529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.103506    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 19:34:25.143165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.143137    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgdz\" (UniqueName: \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.143323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.143210    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.143371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.143311    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.148571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.148543    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"]
Nov 03 19:34:25.149337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.149314    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:34:25.149599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.149428    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:34:25.149599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.149468    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:34:25.149729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.149706    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:34:25.152847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.152826    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"]
Nov 03 19:34:25.152970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.152921    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.207400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.207358    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"]
Nov 03 19:34:25.244510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244479    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.244677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244522    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.244677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244634    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.244762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244682    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.244762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244750    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfrq9\" (UniqueName: \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.244919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.244897    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgdz\" (UniqueName: \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.245221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.245200    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.246986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.246960    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.302809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.302765    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgdz\" (UniqueName: \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\") pod \"maas-api-revoke-sub-770a030d-c2cw9\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.346538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.346465    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.346538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.346516    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.346769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.346547    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfrq9\" (UniqueName: \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.347238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.347213    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.348682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.348661    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.404011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.403984    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfrq9\" (UniqueName: \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\") pod \"maas-api-revoke-sub-344fc6da-lpnvp\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.415896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.415403    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:25.463572 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.463536    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.470727 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.470705    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:25.710521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.710441    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:25.789832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.789800    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"]
Nov 03 19:34:25.794903 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:34:25.794875    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae213f75_435d_4d82_98b3_4dd6dbd78387.slice/crio-00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e WatchSource:0}: Error finding container 00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e: Status 404 returned error can't find the container with id 00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e
Nov 03 19:34:25.847760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.847727    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"]
Nov 03 19:34:25.852926 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:34:25.852900    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd60ef5c9_714b_45c6_b418_6f31205635c5.slice/crio-8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8 WatchSource:0}: Error finding container 8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8: Status 404 returned error can't find the container with id 8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8
Nov 03 19:34:25.933497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.933467    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d" path="/var/lib/kubelet/pods/8aa9d4f2-522b-4fe2-abe5-79b1cdc0a55d/volumes"
Nov 03 19:34:25.933918 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.933904    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99011f11-bdc6-42b1-b33c-dff9aa5ed8e8" path="/var/lib/kubelet/pods/99011f11-bdc6-42b1-b33c-dff9aa5ed8e8/volumes"
Nov 03 19:34:25.967055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.966970    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9" event={"ID":"ae213f75-435d-4d82-98b3-4dd6dbd78387","Type":"ContainerStarted","Data":"00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e"}
Nov 03 19:34:25.968265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.968234    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp" event={"ID":"d60ef5c9-714b-45c6-b418-6f31205635c5","Type":"ContainerStarted","Data":"8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8"}
Nov 03 19:34:25.992422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:25.992399    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:26.975213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:26.975122    4944 generic.go:356] "Generic (PLEG): container finished" podID="d60ef5c9-714b-45c6-b418-6f31205635c5" containerID="767e0165516085163e0ff598a08148a157e186580e85f9812f8a94c5a1aa7fb5" exitCode=0
Nov 03 19:34:26.975606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:26.975205    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp" event={"ID":"d60ef5c9-714b-45c6-b418-6f31205635c5","Type":"ContainerDied","Data":"767e0165516085163e0ff598a08148a157e186580e85f9812f8a94c5a1aa7fb5"}
Nov 03 19:34:26.976764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:26.976745    4944 generic.go:356] "Generic (PLEG): container finished" podID="ae213f75-435d-4d82-98b3-4dd6dbd78387" containerID="379d70e1b4d623f89828282ed322201e57b5fd4b9971c983c2b6761dfa44a9fb" exitCode=0
Nov 03 19:34:26.976877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:26.976817    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9" event={"ID":"ae213f75-435d-4d82-98b3-4dd6dbd78387","Type":"ContainerDied","Data":"379d70e1b4d623f89828282ed322201e57b5fd4b9971c983c2b6761dfa44a9fb"}
Nov 03 19:34:28.121664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.121636    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:28.145869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.145840    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:28.177615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.177585    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\") pod \"ae213f75-435d-4d82-98b3-4dd6dbd78387\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") "
Nov 03 19:34:28.177731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.177649    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\") pod \"ae213f75-435d-4d82-98b3-4dd6dbd78387\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") "
Nov 03 19:34:28.177731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.177678    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\" (UniqueName: \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\") pod \"ae213f75-435d-4d82-98b3-4dd6dbd78387\" (UID: \"ae213f75-435d-4d82-98b3-4dd6dbd78387\") "
Nov 03 19:34:28.177984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.177958    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca" pod "ae213f75-435d-4d82-98b3-4dd6dbd78387" (UID: "ae213f75-435d-4d82-98b3-4dd6dbd78387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:28.179643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.179615    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca" pod "ae213f75-435d-4d82-98b3-4dd6dbd78387" (UID: "ae213f75-435d-4d82-98b3-4dd6dbd78387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:28.179837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.179784    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz" pod "ae213f75-435d-4d82-98b3-4dd6dbd78387" (UID: "ae213f75-435d-4d82-98b3-4dd6dbd78387"). InnerVolumeSpecName "kube-api-access-4dgdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:28.278523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278485    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\") pod \"d60ef5c9-714b-45c6-b418-6f31205635c5\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") "
Nov 03 19:34:28.278707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278580    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\" (UniqueName: \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\") pod \"d60ef5c9-714b-45c6-b418-6f31205635c5\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") "
Nov 03 19:34:28.278707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278630    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\") pod \"d60ef5c9-714b-45c6-b418-6f31205635c5\" (UID: \"d60ef5c9-714b-45c6-b418-6f31205635c5\") "
Nov 03 19:34:28.278974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278950    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.279045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278986    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae213f75-435d-4d82-98b3-4dd6dbd78387-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.279045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.279001    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgdz\" (UniqueName: \"kubernetes.io/projected/ae213f75-435d-4d82-98b3-4dd6dbd78387-kube-api-access-4dgdz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.279045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.278944    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca" pod "d60ef5c9-714b-45c6-b418-6f31205635c5" (UID: "d60ef5c9-714b-45c6-b418-6f31205635c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:28.280729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.280697    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca" pod "d60ef5c9-714b-45c6-b418-6f31205635c5" (UID: "d60ef5c9-714b-45c6-b418-6f31205635c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:34:28.280847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.280742    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9" pod "d60ef5c9-714b-45c6-b418-6f31205635c5" (UID: "d60ef5c9-714b-45c6-b418-6f31205635c5"). InnerVolumeSpecName "kube-api-access-mfrq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:28.379581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.379544    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.379581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.379580    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfrq9\" (UniqueName: \"kubernetes.io/projected/d60ef5c9-714b-45c6-b418-6f31205635c5-kube-api-access-mfrq9\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.379784 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.379593    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d60ef5c9-714b-45c6-b418-6f31205635c5-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:28.990539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.990509    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"
Nov 03 19:34:28.990738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.990548    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp" event={"ID":"d60ef5c9-714b-45c6-b418-6f31205635c5","Type":"ContainerDied","Data":"8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8"}
Nov 03 19:34:28.990738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.990599    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8e7876a3983d30273664ed3025553a28e299c5728b2595701a2f75d40c73d8"
Nov 03 19:34:28.991892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.991868    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"
Nov 03 19:34:28.992024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.991874    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9" event={"ID":"ae213f75-435d-4d82-98b3-4dd6dbd78387","Type":"ContainerDied","Data":"00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e"}
Nov 03 19:34:28.992024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:28.991977    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00ed88df74cdecf298be446e52331f4764f520f23732c3a6ac91de950b676d0e"
Nov 03 19:34:29.997830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:29.997779    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing" containerID="cri-o://f5385c30fb04f142b48e6da84db3f313087fec14273f062d6de97706dbe2f0c2" gracePeriod=25
Nov 03 19:34:30.104573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:30.104514    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing" containerID="cri-o://1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb" gracePeriod=25
Nov 03 19:34:30.672513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:30.672482    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:32.263825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:32.263774    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:32.796852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:32.796819    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:32.809829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:32.809805    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:33.026677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:33.026646    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:34:33.961943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:33.961896    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:34.620864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:34.620827    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:35.230611 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:35.230569    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:34:36.077066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.077031    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.105251 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.105217    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:36.105405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.105252    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:36.325705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.325658    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.458005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.457909    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"]
Nov 03 19:34:36.503525 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.503490    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-eb6ef8-p2lgc"]
Nov 03 19:34:36.578890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.578853    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:36.986370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:36.986323    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:37.933609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:37.933575    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fb5ae9-bbfe-4d31-8383-fbea5d6440dd" path="/var/lib/kubelet/pods/22fb5ae9-bbfe-4d31-8383-fbea5d6440dd/volumes"
Nov 03 19:34:41.194597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:41.194538    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:42.264112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:42.264078    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:43.042215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.042179    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:43.262382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.262350    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:34:43.262632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.262606    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy" containerID="cri-o://c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9" gracePeriod=30
Nov 03 19:34:43.286869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.286846    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:43.530911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.530878    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:43.885554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.885525    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 19:34:43.944891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.944862    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\") pod \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") "
Nov 03 19:34:43.945042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.944952    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\" (UniqueName: \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\") pod \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\" (UID: \"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62\") "
Nov 03 19:34:43.945236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.945213    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume" pod "b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" (UID: "b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:43.947113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:43.947088    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr" pod "b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" (UID: "b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62"). InnerVolumeSpecName "kube-api-access-zwnrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:44.046294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.046261    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnrr\" (UniqueName: \"kubernetes.io/projected/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-kube-api-access-zwnrr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:44.046294 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.046290    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:44.086104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.086015    4944 generic.go:356] "Generic (PLEG): container finished" podID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerID="dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8" exitCode=0
Nov 03 19:34:44.086104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.086068    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"
Nov 03 19:34:44.086323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.086092    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" event={"ID":"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62","Type":"ContainerDied","Data":"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"}
Nov 03 19:34:44.086323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.086161    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz" event={"ID":"b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62","Type":"ContainerDied","Data":"26a8c42fc38df937eb8c6de9e8935720faf6f0f48e9ae81a71cc29b208f65e65"}
Nov 03 19:34:44.086323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.086191    4944 scope.go:122] "RemoveContainer" containerID="dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"
Nov 03 19:34:44.097325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.097289    4944 scope.go:122] "RemoveContainer" containerID="dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"
Nov 03 19:34:44.097700 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:44.097613    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8\": container with ID starting with dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8 not found: ID does not exist" containerID="dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"
Nov 03 19:34:44.097700 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.097670    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8"} err="failed to get container status \"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8\": rpc error: code = NotFound desc = could not find container \"dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8\": container with ID starting with dcd27a1494b9037b0846460c89f8c09f99296b6434ec3e034032f44544adcab8 not found: ID does not exist"
Nov 03 19:34:44.239677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.239647    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 19:34:44.243832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:44.243807    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-eb6ef8-6495544574-f6rfz"]
Nov 03 19:34:45.012178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:45.012142    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:45.260770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:45.260736    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:45.752966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:45.752932    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:45.935668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:45.935634    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" path="/var/lib/kubelet/pods/b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62/volumes"
Nov 03 19:34:46.001043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:46.001012    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:46.054474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:46.054396    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"]
Nov 03 19:34:46.100338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:46.100308    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b44dc7-gwfkm"]
Nov 03 19:34:46.105407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:46.105373    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:46.105540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:46.105395    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:34:47.934942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:47.934898    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d272c3e4-c581-4594-89ba-0e4593b96fb1" path="/var/lib/kubelet/pods/d272c3e4-c581-4594-89ba-0e4593b96fb1/volumes"
Nov 03 19:34:48.516033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.516007    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:34:48.589888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589780    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.589888 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589848    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589889    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589917    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589958    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.589986    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590085    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590090    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:48.590136 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590114    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590160    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\") pod \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\" (UID: \"af168f41-4e76-4cc9-bf50-e2e9f0a7a125\") "
Nov 03 19:34:48.590506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590237    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:48.590506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590335    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:48.590506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590440    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:48.590733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590607    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.590733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590631    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.590733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590645    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.590733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590659    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.590733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.590656    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:48.592111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.592080    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:34:48.592275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.592257    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:34:48.592526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.592504    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:48.592618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.592530    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc" pod "af168f41-4e76-4cc9-bf50-e2e9f0a7a125" (UID: "af168f41-4e76-4cc9-bf50-e2e9f0a7a125"). InnerVolumeSpecName "kube-api-access-k6jvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:48.691670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.691640    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.691670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.691666    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6jvc\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-kube-api-access-k6jvc\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.691670 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.691676    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.691916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.691686    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:48.691916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:48.691695    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af168f41-4e76-4cc9-bf50-e2e9f0a7a125-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:49.126739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.126707    4944 generic.go:356] "Generic (PLEG): container finished" podID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerID="c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9" exitCode=0
Nov 03 19:34:49.127153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.126759    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"
Nov 03 19:34:49.127153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.126799    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" event={"ID":"af168f41-4e76-4cc9-bf50-e2e9f0a7a125","Type":"ContainerDied","Data":"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"}
Nov 03 19:34:49.127153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.126855    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9" event={"ID":"af168f41-4e76-4cc9-bf50-e2e9f0a7a125","Type":"ContainerDied","Data":"6824344a9f706ed4dd79fea0ace86e047563c897fafc4997c6df13eb3b0333be"}
Nov 03 19:34:49.127153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.126877    4944 scope.go:122] "RemoveContainer" containerID="c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"
Nov 03 19:34:49.137314 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.137293    4944 scope.go:122] "RemoveContainer" containerID="c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"
Nov 03 19:34:49.137561 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:49.137543    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9\": container with ID starting with c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9 not found: ID does not exist" containerID="c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"
Nov 03 19:34:49.137606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.137569    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9"} err="failed to get container status \"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9\": rpc error: code = NotFound desc = could not find container \"c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9\": container with ID starting with c86c17574a70dbe73bff2a250fd116d5e28a27cf5407b7e54e519833c40e4df9 not found: ID does not exist"
Nov 03 19:34:49.280802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.280755    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:34:49.285016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.284991    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-eb6ef8-openshift-default-5669588b8f-xbqw9"]
Nov 03 19:34:49.787267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.787231    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:49.933651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:49.933615    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" path="/var/lib/kubelet/pods/af168f41-4e76-4cc9-bf50-e2e9f0a7a125/volumes"
Nov 03 19:34:50.135989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.135963    4944 generic.go:356] "Generic (PLEG): container finished" podID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerID="f5385c30fb04f142b48e6da84db3f313087fec14273f062d6de97706dbe2f0c2" exitCode=0
Nov 03 19:34:50.136321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.136033    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" event={"ID":"f58b2f48-552b-4467-ae5c-09572c5200d4","Type":"ContainerDied","Data":"f5385c30fb04f142b48e6da84db3f313087fec14273f062d6de97706dbe2f0c2"}
Nov 03 19:34:50.258863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.258828    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 19:34:50.314832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.314770    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\") pod \"f58b2f48-552b-4467-ae5c-09572c5200d4\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") "
Nov 03 19:34:50.315024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.314875    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\" (UniqueName: \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\") pod \"f58b2f48-552b-4467-ae5c-09572c5200d4\" (UID: \"f58b2f48-552b-4467-ae5c-09572c5200d4\") "
Nov 03 19:34:50.315933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.315901    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume" pod "f58b2f48-552b-4467-ae5c-09572c5200d4" (UID: "f58b2f48-552b-4467-ae5c-09572c5200d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:50.317830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.317753    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt" pod "f58b2f48-552b-4467-ae5c-09572c5200d4" (UID: "f58b2f48-552b-4467-ae5c-09572c5200d4"). InnerVolumeSpecName "kube-api-access-9lpmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:50.357643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.357619    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 19:34:50.416175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.416100    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\" (UniqueName: \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\") pod \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") "
Nov 03 19:34:50.416321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.416193    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\") pod \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\" (UID: \"aef37501-ac5d-4cfe-a36a-b9e8a3757148\") "
Nov 03 19:34:50.416449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.416435    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lpmt\" (UniqueName: \"kubernetes.io/projected/f58b2f48-552b-4467-ae5c-09572c5200d4-kube-api-access-9lpmt\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:50.416497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.416451    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58b2f48-552b-4467-ae5c-09572c5200d4-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:50.416541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.416517    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume" pod "aef37501-ac5d-4cfe-a36a-b9e8a3757148" (UID: "aef37501-ac5d-4cfe-a36a-b9e8a3757148"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:34:50.418102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.418074    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z" pod "aef37501-ac5d-4cfe-a36a-b9e8a3757148" (UID: "aef37501-ac5d-4cfe-a36a-b9e8a3757148"). InnerVolumeSpecName "kube-api-access-k486z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:34:50.517043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.517011    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aef37501-ac5d-4cfe-a36a-b9e8a3757148-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:50.517043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:50.517040    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k486z\" (UniqueName: \"kubernetes.io/projected/aef37501-ac5d-4cfe-a36a-b9e8a3757148-kube-api-access-k486z\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:34:51.143265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.143226    4944 generic.go:356] "Generic (PLEG): container finished" podID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerID="1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb" exitCode=0
Nov 03 19:34:51.143714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.143281    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"
Nov 03 19:34:51.143714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.143314    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" event={"ID":"aef37501-ac5d-4cfe-a36a-b9e8a3757148","Type":"ContainerDied","Data":"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"}
Nov 03 19:34:51.143714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.143371    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq" event={"ID":"aef37501-ac5d-4cfe-a36a-b9e8a3757148","Type":"ContainerDied","Data":"a86318e24e84d3322205329c00f54ed1fe8dab01429622628e0ffff90db22911"}
Nov 03 19:34:51.143714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.143392    4944 scope.go:122] "RemoveContainer" containerID="1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"
Nov 03 19:34:51.145160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.145138    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"
Nov 03 19:34:51.145269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.145130    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl" event={"ID":"f58b2f48-552b-4467-ae5c-09572c5200d4","Type":"ContainerDied","Data":"9b7fe176d4bbee6ab1c17aca56499fc4eec8cdbbdb2458164a601eee994a4d25"}
Nov 03 19:34:51.153268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.153237    4944 scope.go:122] "RemoveContainer" containerID="1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"
Nov 03 19:34:51.153545 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:34:51.153515    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb\": container with ID starting with 1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb not found: ID does not exist" containerID="1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"
Nov 03 19:34:51.153660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.153557    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb"} err="failed to get container status \"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb\": rpc error: code = NotFound desc = could not find container \"1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb\": container with ID starting with 1adb43711e9503c52b4aa519d70463d3e4fc912e0187aebda90bb5fbff0cecbb not found: ID does not exist"
Nov 03 19:34:51.153660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.153580    4944 scope.go:122] "RemoveContainer" containerID="f5385c30fb04f142b48e6da84db3f313087fec14273f062d6de97706dbe2f0c2"
Nov 03 19:34:51.269375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.269267    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:51.296290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.296262    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 19:34:51.342589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.342559    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-fa2878-7c747f89ff-9msqq"]
Nov 03 19:34:51.403681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.403649    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 19:34:51.409894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.409869    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-fa2878-655774c49b-nsszl"]
Nov 03 19:34:51.516764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.516725    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:51.633738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.633657    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"]
Nov 03 19:34:51.639651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.639626    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-fa2878-w9w2r"]
Nov 03 19:34:51.763882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.763853    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:51.936974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.936885    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" path="/var/lib/kubelet/pods/aef37501-ac5d-4cfe-a36a-b9e8a3757148/volumes"
Nov 03 19:34:51.937528 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.937508    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfca82d4-94c6-464f-83c6-d171ccf92892" path="/var/lib/kubelet/pods/cfca82d4-94c6-464f-83c6-d171ccf92892/volumes"
Nov 03 19:34:51.938063 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:51.938043    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" path="/var/lib/kubelet/pods/f58b2f48-552b-4467-ae5c-09572c5200d4/volumes"
Nov 03 19:34:52.012683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:52.012654    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:52.257697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:52.257666    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:52.503553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:52.503523    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:56.464005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:56.463968    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:58.824381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:58.824350    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:59.074653 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:59.074551    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:34:59.325479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:34:59.325391    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:01.071176 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:01.071143    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:02.394479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:02.394453    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:35:02.396910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:02.396890    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:35:05.223845 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:35:05.223784    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:35:05.254703 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:05.254676    4944 scope.go:122] "RemoveContainer" containerID="fd9183d2d7ed9cdeb0dbf6a2a099585ecb469a557e4cb915039a3cef11fa2495"
Nov 03 19:35:05.263575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:05.263554    4944 scope.go:122] "RemoveContainer" containerID="6451742ff7e151d62a3a4f169f4a97b0e6d5c388bc5380de7a3f28f29f5b0453"
Nov 03 19:35:05.271694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:05.271672    4944 scope.go:122] "RemoveContainer" containerID="e0df815b604af5fe5e339e6b61f870acc47302ffc37d04b2841671ede9675098"
Nov 03 19:35:05.764743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:05.764698    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:10.481956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:10.481910    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:14.046086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:14.046009    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"]
Nov 03 19:35:14.092283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:14.092248    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53927318-m77xr"]
Nov 03 19:35:15.074183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:15.074150    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:15.933508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:15.933474    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a19da2f-0cda-4463-9856-e8be2dd69c44" path="/var/lib/kubelet/pods/2a19da2f-0cda-4463-9856-e8be2dd69c44/volumes"
Nov 03 19:35:19.712240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:19.712193    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:21.806317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:21.806270    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:22.056495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:22.056413    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:22.783846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:22.783813    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:24.247967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:24.247928    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:26.856017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:26.855983    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:28.856931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:28.856900    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:33.418558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:33.418518    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:34.282984 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:34.282948    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:34.616498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:34.616419    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:35.224263 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:35:35.224227    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:35:38.167581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:38.167547    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:42.640146 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:42.640100    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:47.157284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:47.157226    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:48.068916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.068879    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"]
Nov 03 19:35:48.069322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069308    4944 cpu_manager.go:414] "containerMap: removing container" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069324    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069337    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069342    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069349    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ae213f75-435d-4d82-98b3-4dd6dbd78387" containerName="revoke-keys"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069355    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae213f75-435d-4d82-98b3-4dd6dbd78387" containerName="revoke-keys"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069369    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing"
Nov 03 19:35:48.069375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069374    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069382    4944 cpu_manager.go:414] "containerMap: removing container" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069386    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069393    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d60ef5c9-714b-45c6-b418-6f31205635c5" containerName="revoke-keys"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069397    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60ef5c9-714b-45c6-b418-6f31205635c5" containerName="revoke-keys"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069468    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="aef37501-ac5d-4cfe-a36a-b9e8a3757148" containerName="payload-pre-processing"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069475    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58b2f48-552b-4467-ae5c-09572c5200d4" containerName="payload-processing"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069483    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae213f75-435d-4d82-98b3-4dd6dbd78387" containerName="revoke-keys"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069491    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d0db6f-bb1a-4fc0-aadc-6c6f88396b62" containerName="payload-processing"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069497    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="af168f41-4e76-4cc9-bf50-e2e9f0a7a125" containerName="istio-proxy"
Nov 03 19:35:48.069597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.069502    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60ef5c9-714b-45c6-b418-6f31205635c5" containerName="revoke-keys"
Nov 03 19:35:48.072756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.072736    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.127080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.127050    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"]
Nov 03 19:35:48.170414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.170383    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:35:48.170946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.170441    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:35:48.170946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.170491    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:35:48.170946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.170487    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:35:48.184655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.184628    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.184815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.184690    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xdlm\" (UniqueName: \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.184880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.184859    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.285597 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.285561    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.285829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.285631    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xdlm\" (UniqueName: \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.285829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.285682    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.286317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.286288    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.288068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.288048    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.343006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.342941    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xdlm\" (UniqueName: \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\") pod \"maas-api-revoke-sub-82f98dde-lvmqd\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.385819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.385774    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:48.755771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:48.755739    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"]
Nov 03 19:35:48.761533 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:35:48.761504    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57de78a5_24e4_4789_ac31_fa31eed49dfa.slice/crio-ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407 WatchSource:0}: Error finding container ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407: Status 404 returned error can't find the container with id ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407
Nov 03 19:35:49.494272 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:49.494234    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd" event={"ID":"57de78a5-24e4-4789-ac31-fa31eed49dfa","Type":"ContainerStarted","Data":"c97f7d57789fa1e6dfe8e9f3991faf1510899859d1e677ae0e9e199fd0b8e75c"}
Nov 03 19:35:49.494744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:49.494290    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd" event={"ID":"57de78a5-24e4-4789-ac31-fa31eed49dfa","Type":"ContainerStarted","Data":"ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407"}
Nov 03 19:35:49.603636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:49.603588    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd" podStartSLOduration=1.6035756380000001 podStartE2EDuration="1.603575638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:35:49.601973917 +0000 UTC m=+3047.879327501" watchObservedRunningTime="2026-11-03 19:35:49.603575638 +0000 UTC m=+3047.880929221"
Nov 03 19:35:50.501526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:50.501492    4944 generic.go:356] "Generic (PLEG): container finished" podID="57de78a5-24e4-4789-ac31-fa31eed49dfa" containerID="c97f7d57789fa1e6dfe8e9f3991faf1510899859d1e677ae0e9e199fd0b8e75c" exitCode=0
Nov 03 19:35:50.501962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:50.501574    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd" event={"ID":"57de78a5-24e4-4789-ac31-fa31eed49dfa","Type":"ContainerDied","Data":"c97f7d57789fa1e6dfe8e9f3991faf1510899859d1e677ae0e9e199fd0b8e75c"}
Nov 03 19:35:51.642405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.642373    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:51.723556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.723519    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\" (UniqueName: \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\") pod \"57de78a5-24e4-4789-ac31-fa31eed49dfa\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") "
Nov 03 19:35:51.723734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.723567    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\") pod \"57de78a5-24e4-4789-ac31-fa31eed49dfa\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") "
Nov 03 19:35:51.723734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.723597    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\") pod \"57de78a5-24e4-4789-ac31-fa31eed49dfa\" (UID: \"57de78a5-24e4-4789-ac31-fa31eed49dfa\") "
Nov 03 19:35:51.724131 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.724103    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca" pod "57de78a5-24e4-4789-ac31-fa31eed49dfa" (UID: "57de78a5-24e4-4789-ac31-fa31eed49dfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:35:51.725737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.725706    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca" pod "57de78a5-24e4-4789-ac31-fa31eed49dfa" (UID: "57de78a5-24e4-4789-ac31-fa31eed49dfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:35:51.725994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.725976    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm" pod "57de78a5-24e4-4789-ac31-fa31eed49dfa" (UID: "57de78a5-24e4-4789-ac31-fa31eed49dfa"). InnerVolumeSpecName "kube-api-access-7xdlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:35:51.739996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.739974    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:51.824941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.824870    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xdlm\" (UniqueName: \"kubernetes.io/projected/57de78a5-24e4-4789-ac31-fa31eed49dfa-kube-api-access-7xdlm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:51.824941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.824899    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:51.824941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:51.824912    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57de78a5-24e4-4789-ac31-fa31eed49dfa-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:35:52.518256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:52.518224    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"
Nov 03 19:35:52.518448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:52.518219    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd" event={"ID":"57de78a5-24e4-4789-ac31-fa31eed49dfa","Type":"ContainerDied","Data":"ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407"}
Nov 03 19:35:52.518448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:52.518356    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ead2369e355c3aa949d8a85abd6e1eb1c496686b625769c5c20f3f91b09af407"
Nov 03 19:35:54.923344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:54.923301    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:56.306472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:56.306432    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:35:59.992744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:35:59.992702    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:36:00.107883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.107818    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:00.863836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.863778    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:00.886998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.886962    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"]
Nov 03 19:36:00.887473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.887452    4944 cpu_manager.go:414] "containerMap: removing container" podUID="57de78a5-24e4-4789-ac31-fa31eed49dfa" containerName="revoke-keys"
Nov 03 19:36:00.887565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.887474    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57de78a5-24e4-4789-ac31-fa31eed49dfa" containerName="revoke-keys"
Nov 03 19:36:00.887629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.887591    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="57de78a5-24e4-4789-ac31-fa31eed49dfa" containerName="revoke-keys"
Nov 03 19:36:00.890689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.890666    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:00.942261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.942227    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:36:00.985412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.985381    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:36:00.985412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.985402    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:36:00.985605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.985585    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:36:00.987881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:00.987864    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"]
Nov 03 19:36:01.021219 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.021193    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdn2m\" (UniqueName: \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.021584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.021224    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.021584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.021263    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.122586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.122508    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdn2m\" (UniqueName: \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.122586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.122545    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.122586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.122583    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.123361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.123337    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.125104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.125079    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.179233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.179206    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdn2m\" (UniqueName: \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\") pod \"maas-api-revoke-sub-2afa005b-m5rmw\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.203664 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.203641    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:01.571290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:01.571259    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"]
Nov 03 19:36:01.578137 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:36:01.578111    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47159d18_da63_4c6a_b1fc_26b5a26cda6c.slice/crio-7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc WatchSource:0}: Error finding container 7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc: Status 404 returned error can't find the container with id 7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc
Nov 03 19:36:02.576291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:02.576257    4944 generic.go:356] "Generic (PLEG): container finished" podID="47159d18-da63-4c6a-b1fc-26b5a26cda6c" containerID="13df82360fadd2ae05c86dea4dd1498cde2e4231146b18494f870053e569e6f1" exitCode=0
Nov 03 19:36:02.576732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:02.576342    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw" event={"ID":"47159d18-da63-4c6a-b1fc-26b5a26cda6c","Type":"ContainerDied","Data":"13df82360fadd2ae05c86dea4dd1498cde2e4231146b18494f870053e569e6f1"}
Nov 03 19:36:02.576732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:02.576394    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw" event={"ID":"47159d18-da63-4c6a-b1fc-26b5a26cda6c","Type":"ContainerStarted","Data":"7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc"}
Nov 03 19:36:03.720661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.720633    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:03.744423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.744399    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\") pod \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") "
Nov 03 19:36:03.744590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.744440    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\" (UniqueName: \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\") pod \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") "
Nov 03 19:36:03.744590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.744567    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\") pod \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\" (UID: \"47159d18-da63-4c6a-b1fc-26b5a26cda6c\") "
Nov 03 19:36:03.744832 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.744785    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca" pod "47159d18-da63-4c6a-b1fc-26b5a26cda6c" (UID: "47159d18-da63-4c6a-b1fc-26b5a26cda6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:36:03.745023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.745005    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:03.746506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.746485    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m" pod "47159d18-da63-4c6a-b1fc-26b5a26cda6c" (UID: "47159d18-da63-4c6a-b1fc-26b5a26cda6c"). InnerVolumeSpecName "kube-api-access-pdn2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:03.747012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.746990    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca" pod "47159d18-da63-4c6a-b1fc-26b5a26cda6c" (UID: "47159d18-da63-4c6a-b1fc-26b5a26cda6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:36:03.845561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.845492    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdn2m\" (UniqueName: \"kubernetes.io/projected/47159d18-da63-4c6a-b1fc-26b5a26cda6c-kube-api-access-pdn2m\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:03.845561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:03.845519    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47159d18-da63-4c6a-b1fc-26b5a26cda6c-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:04.589447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:04.589409    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw" event={"ID":"47159d18-da63-4c6a-b1fc-26b5a26cda6c","Type":"ContainerDied","Data":"7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc"}
Nov 03 19:36:04.589447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:04.589432    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"
Nov 03 19:36:04.589665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:04.589460    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7392dde3442c359456f249bd7a98c38d43c40783d05fe4475e08f562856fabdc"
Nov 03 19:36:05.224147 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:36:05.224108    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:36:05.323579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:05.323547    4944 scope.go:122] "RemoveContainer" containerID="07ea36cb063dbb05ab2646bb8b88499904500eaa678d4e2d22c1a9c7940d8af4"
Nov 03 19:36:05.396939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:05.396909    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:09.899507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:09.899468    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:14.453757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:14.453724    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:15.012359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.012298    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" podUID="9e23bda4-c7a2-4f96-8410-b59b46b89069" containerName="main" containerID="cri-o://5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3" gracePeriod=45
Nov 03 19:36:15.263802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.263729    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:36:15.371805 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.371767    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.372009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.371828    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\" (UniqueName: \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.372009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.371869    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.372009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.371913    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.372009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.371985    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.373082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.372015    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\") pod \"9e23bda4-c7a2-4f96-8410-b59b46b89069\" (UID: \"9e23bda4-c7a2-4f96-8410-b59b46b89069\") "
Nov 03 19:36:15.373082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.372247    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:15.373082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.372438    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:15.373082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.372450    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:15.373082 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.372450    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.374877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.374848    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:36:15.375014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.374989    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "kube-api-access-chl7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:15.376485 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.376462    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm" pod "9e23bda4-c7a2-4f96-8410-b59b46b89069" (UID: "9e23bda4-c7a2-4f96-8410-b59b46b89069"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:15.473223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.473188    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.473223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.473219    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.473223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.473230    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e23bda4-c7a2-4f96-8410-b59b46b89069-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.473667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.473239    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chl7j\" (UniqueName: \"kubernetes.io/projected/9e23bda4-c7a2-4f96-8410-b59b46b89069-kube-api-access-chl7j\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.473667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.473248    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e23bda4-c7a2-4f96-8410-b59b46b89069-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:15.653949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.653863    4944 generic.go:356] "Generic (PLEG): container finished" podID="9e23bda4-c7a2-4f96-8410-b59b46b89069" containerID="5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3" exitCode=0
Nov 03 19:36:15.653949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.653902    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" event={"ID":"9e23bda4-c7a2-4f96-8410-b59b46b89069","Type":"ContainerDied","Data":"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"}
Nov 03 19:36:15.653949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.653916    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"
Nov 03 19:36:15.653949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.653933    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk" event={"ID":"9e23bda4-c7a2-4f96-8410-b59b46b89069","Type":"ContainerDied","Data":"75384942c60f10bd4983db7217015a279fdf1d5834c805aa893d7d2aa7f238cb"}
Nov 03 19:36:15.653949 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.653950    4944 scope.go:122] "RemoveContainer" containerID="5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"
Nov 03 19:36:15.663910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.663891    4944 scope.go:122] "RemoveContainer" containerID="5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"
Nov 03 19:36:15.664169 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:36:15.664149    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3\": container with ID starting with 5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3 not found: ID does not exist" containerID="5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"
Nov 03 19:36:15.664234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.664181    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3"} err="failed to get container status \"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3\": rpc error: code = NotFound desc = could not find container \"5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3\": container with ID starting with 5a5d0a62cd1b206fa4fc4ce44b313f6e20f9be3bf44ceb8140d5833017bf9dc3 not found: ID does not exist"
Nov 03 19:36:15.806910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.806866    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:36:15.853152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.853119    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fcb646/test-model-fcb646-kserve-5bd99c749-48vvk"]
Nov 03 19:36:15.934355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:15.934266    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e23bda4-c7a2-4f96-8410-b59b46b89069" path="/var/lib/kubelet/pods/9e23bda4-c7a2-4f96-8410-b59b46b89069/volumes"
Nov 03 19:36:19.093569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:19.093535    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:23.475980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:23.475945    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:27.792104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:27.792068    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:36:27.911970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:27.911909    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:28.064672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.064590    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"]
Nov 03 19:36:28.065222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065200    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9e23bda4-c7a2-4f96-8410-b59b46b89069" containerName="main"
Nov 03 19:36:28.065282 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065225    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e23bda4-c7a2-4f96-8410-b59b46b89069" containerName="main"
Nov 03 19:36:28.065327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065281    4944 cpu_manager.go:414] "containerMap: removing container" podUID="47159d18-da63-4c6a-b1fc-26b5a26cda6c" containerName="revoke-keys"
Nov 03 19:36:28.065327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065290    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47159d18-da63-4c6a-b1fc-26b5a26cda6c" containerName="revoke-keys"
Nov 03 19:36:28.065407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065395    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="47159d18-da63-4c6a-b1fc-26b5a26cda6c" containerName="revoke-keys"
Nov 03 19:36:28.065452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.065411    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e23bda4-c7a2-4f96-8410-b59b46b89069" containerName="main"
Nov 03 19:36:28.070478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.070456    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.119989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.119955    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:36:28.119989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.119958    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:36:28.119989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.119960    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:36:28.120289 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.119958    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:36:28.122829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.122776    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"]
Nov 03 19:36:28.168253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.168218    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:28.200192 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.200163    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.200393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.200212    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdm28\" (UniqueName: \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.200393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.200298    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.300938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.300898    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdm28\" (UniqueName: \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.301140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.300959    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.301140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.301089    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.301694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.301671    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.303320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.303300    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.358092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.358025    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdm28\" (UniqueName: \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\") pod \"maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.383686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.383658    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:28.752260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:28.752228    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"]
Nov 03 19:36:28.759012 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:36:28.758987    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5005b91_ce60_4245_8beb_eec4cf2abcb0.slice/crio-bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e WatchSource:0}: Error finding container bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e: Status 404 returned error can't find the container with id bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e
Nov 03 19:36:29.740741 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:29.740703    4944 generic.go:356] "Generic (PLEG): container finished" podID="e5005b91-ce60-4245-8beb-eec4cf2abcb0" containerID="a92f593e14856b7b78781c860828d6e0b92b31dd103e2f71d89658bb96d54d40" exitCode=0
Nov 03 19:36:29.741124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:29.740807    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf" event={"ID":"e5005b91-ce60-4245-8beb-eec4cf2abcb0","Type":"ContainerDied","Data":"a92f593e14856b7b78781c860828d6e0b92b31dd103e2f71d89658bb96d54d40"}
Nov 03 19:36:29.741124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:29.740860    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf" event={"ID":"e5005b91-ce60-4245-8beb-eec4cf2abcb0","Type":"ContainerStarted","Data":"bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e"}
Nov 03 19:36:30.888696 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:30.888672    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:31.028535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.028457    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\") pod \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") "
Nov 03 19:36:31.028807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.028547    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\") pod \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") "
Nov 03 19:36:31.028807 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.028611    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\" (UniqueName: \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\") pod \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\" (UID: \"e5005b91-ce60-4245-8beb-eec4cf2abcb0\") "
Nov 03 19:36:31.028902 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.028836    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca" pod "e5005b91-ce60-4245-8beb-eec4cf2abcb0" (UID: "e5005b91-ce60-4245-8beb-eec4cf2abcb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:36:31.029075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.029036    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:31.030958 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.030930    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28" pod "e5005b91-ce60-4245-8beb-eec4cf2abcb0" (UID: "e5005b91-ce60-4245-8beb-eec4cf2abcb0"). InnerVolumeSpecName "kube-api-access-pdm28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:31.031231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.031211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca" pod "e5005b91-ce60-4245-8beb-eec4cf2abcb0" (UID: "e5005b91-ce60-4245-8beb-eec4cf2abcb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:36:31.130123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.130087    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5005b91-ce60-4245-8beb-eec4cf2abcb0-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:31.130123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.130119    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdm28\" (UniqueName: \"kubernetes.io/projected/e5005b91-ce60-4245-8beb-eec4cf2abcb0-kube-api-access-pdm28\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:31.752730 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.752688    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"
Nov 03 19:36:31.752933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.752726    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf" event={"ID":"e5005b91-ce60-4245-8beb-eec4cf2abcb0","Type":"ContainerDied","Data":"bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e"}
Nov 03 19:36:31.752933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:31.752776    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bebdaa5f8d0d64271f0bc9ae7a55b9b8bf074996be6b469243c83cd96174488e"
Nov 03 19:36:33.164960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:33.164922    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:34.628576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:34.628534    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:35.223769 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:36:35.223732    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:36:37.637596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:37.637558    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:39.674330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:39.674290    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:36:39.774916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:39.774879    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:36:40.161521 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:40.161489    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:40.413344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:40.413260    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:42.269738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:42.269664    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:42.811870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:42.811821    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerName="main" containerID="cri-o://3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c" gracePeriod=45
Nov 03 19:36:43.061609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.061587    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:36:43.128862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.128762    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.128862 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.128841    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.129096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.128870    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.129096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.128912    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\" (UniqueName: \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.129096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.128953    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.129096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129055    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\") pod \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\" (UID: \"2ca30fb7-2245-4353-b16c-c8cbe5389e7f\") "
Nov 03 19:36:43.129327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129142    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:43.129327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129154    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:43.129441 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129370    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:43.129498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129471    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.129498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129490    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.129589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.129504    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.131096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.131071    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:36:43.131096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.131088    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "kube-api-access-d9g9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:36:43.131096 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.131079    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs" pod "2ca30fb7-2245-4353-b16c-c8cbe5389e7f" (UID: "2ca30fb7-2245-4353-b16c-c8cbe5389e7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:36:43.230864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.230832    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9g9d\" (UniqueName: \"kubernetes.io/projected/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-kube-api-access-d9g9d\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.230864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.230858    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.230864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.230868    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ca30fb7-2245-4353-b16c-c8cbe5389e7f-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:36:43.824043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.824007    4944 generic.go:356] "Generic (PLEG): container finished" podID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerID="3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c" exitCode=0
Nov 03 19:36:43.824452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.824067    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"
Nov 03 19:36:43.824452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.824088    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" event={"ID":"2ca30fb7-2245-4353-b16c-c8cbe5389e7f","Type":"ContainerDied","Data":"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"}
Nov 03 19:36:43.824452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.824137    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" event={"ID":"2ca30fb7-2245-4353-b16c-c8cbe5389e7f","Type":"ContainerDied","Data":"86a2dca3b0ca9b64aff10337f83843cbab61800d717c49ffb8be3e260f53f153"}
Nov 03 19:36:43.824452 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.824159    4944 scope.go:122] "RemoveContainer" containerID="3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"
Nov 03 19:36:43.834194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.834178    4944 scope.go:122] "RemoveContainer" containerID="3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"
Nov 03 19:36:43.834450 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:36:43.834431    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c\": container with ID starting with 3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c not found: ID does not exist" containerID="3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"
Nov 03 19:36:43.834522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.834458    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c"} err="failed to get container status \"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c\": rpc error: code = NotFound desc = could not find container \"3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c\": container with ID starting with 3f56e18fec3edb7b05824dd0bdbd215e4abe79de857b696b17a3ed000c94614c not found: ID does not exist"
Nov 03 19:36:43.977812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:43.977762    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:36:44.016917 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:44.016868    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerName="main" probeResult="failure" output="Get \"https://10.134.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:36:44.024035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:44.024005    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-24cdad/test-model-24cdad-kserve-78b7bbd56c-xzwlv"]
Nov 03 19:36:44.675582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:44.675531    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing" containerID="cri-o://7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08" gracePeriod=25
Nov 03 19:36:44.775169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:44.775126    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing" containerID="cri-o://1f63272dfae63ad534fda8c1eefcf9039314828038a415a6a093b9e985c2552b" gracePeriod=25
Nov 03 19:36:45.933149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:45.933114    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" path="/var/lib/kubelet/pods/2ca30fb7-2245-4353-b16c-c8cbe5389e7f/volumes"
Nov 03 19:36:46.736691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:46.736652    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:46.971201 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:46.971165    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:36:46.971600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:46.971230    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:36:50.800464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:50.800405    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:51.371212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:51.371177    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.236594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:52.236561    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.483138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:52.483095    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.731285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:52.731248    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"]
Nov 03 19:36:52.737463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:52.737431    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:52.777511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:52.777484    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fcb646-krtnf"]
Nov 03 19:36:53.267960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:53.267925    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:53.932660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:53.932627    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5005b91-ce60-4245-8beb-eec4cf2abcb0" path="/var/lib/kubelet/pods/e5005b91-ce60-4245-8beb-eec4cf2abcb0/volumes"
Nov 03 19:36:56.971556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:56.971520    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:36:56.971970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:56.971588    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:36:57.252620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:57.252588    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:59.043004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.042967    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"]
Nov 03 19:36:59.089235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.089202    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d3e09b9-wgn4q"]
Nov 03 19:36:59.240256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.240224    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:59.454569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.454493    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:36:59.454777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.454753    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy" containerID="cri-o://d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb" gracePeriod=30
Nov 03 19:36:59.489857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.489823    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:59.740563 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.740531    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:36:59.932604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:36:59.932567    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f" path="/var/lib/kubelet/pods/7d8e9d57-72e8-4552-a2a6-68f5ecaf7c0f/volumes"
Nov 03 19:37:00.054627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.054546    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"]
Nov 03 19:37:00.055033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.054986    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e5005b91-ce60-4245-8beb-eec4cf2abcb0" containerName="revoke-keys"
Nov 03 19:37:00.055033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.054998    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5005b91-ce60-4245-8beb-eec4cf2abcb0" containerName="revoke-keys"
Nov 03 19:37:00.055033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.055017    4944 cpu_manager.go:414] "containerMap: removing container" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerName="main"
Nov 03 19:37:00.055033 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.055022    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerName="main"
Nov 03 19:37:00.055196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.055089    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5005b91-ce60-4245-8beb-eec4cf2abcb0" containerName="revoke-keys"
Nov 03 19:37:00.055196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.055100    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca30fb7-2245-4353-b16c-c8cbe5389e7f" containerName="main"
Nov 03 19:37:00.058057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.058036    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.109770 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.109694    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:37:00.110065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.110035    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:37:00.110186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.110148    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:37:00.110257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.110211    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:37:00.113144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.113118    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"]
Nov 03 19:37:00.186352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.186321    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.186522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.186463    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.186565 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.186518    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wk2z\" (UniqueName: \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.287709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.287676    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.287892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.287715    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wk2z\" (UniqueName: \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.287950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.287893    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.288511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.288487    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.290123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.290104    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.344864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.344783    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wk2z\" (UniqueName: \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\") pod \"maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.370750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.370724    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:00.745440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.745406    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"]
Nov 03 19:37:00.752250 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:37:00.752209    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48124b9d_789a_4c58_9949_9a32c2f65854.slice/crio-bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a WatchSource:0}: Error finding container bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a: Status 404 returned error can't find the container with id bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a
Nov 03 19:37:00.923322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:00.923280    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2" event={"ID":"48124b9d-789a-4c58-9949-9a32c2f65854","Type":"ContainerStarted","Data":"bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a"}
Nov 03 19:37:01.931484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:01.931390    4944 generic.go:356] "Generic (PLEG): container finished" podID="48124b9d-789a-4c58-9949-9a32c2f65854" containerID="0638434e7237524330c9bdd24100d5a34b9491529545e47b051ce8fc79c8f979" exitCode=0
Nov 03 19:37:01.933307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:01.933272    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2" event={"ID":"48124b9d-789a-4c58-9949-9a32c2f65854","Type":"ContainerDied","Data":"0638434e7237524330c9bdd24100d5a34b9491529545e47b051ce8fc79c8f979"}
Nov 03 19:37:02.058625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:02.058590    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:03.074379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.074353    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:03.220910 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.220840    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\") pod \"48124b9d-789a-4c58-9949-9a32c2f65854\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") "
Nov 03 19:37:03.221078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.220912    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\" (UniqueName: \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\") pod \"48124b9d-789a-4c58-9949-9a32c2f65854\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") "
Nov 03 19:37:03.221078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.220968    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\") pod \"48124b9d-789a-4c58-9949-9a32c2f65854\" (UID: \"48124b9d-789a-4c58-9949-9a32c2f65854\") "
Nov 03 19:37:03.221215 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.221189    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca" pod "48124b9d-789a-4c58-9949-9a32c2f65854" (UID: "48124b9d-789a-4c58-9949-9a32c2f65854"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:03.222962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.222935    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca" pod "48124b9d-789a-4c58-9949-9a32c2f65854" (UID: "48124b9d-789a-4c58-9949-9a32c2f65854"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:03.223075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.223049    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z" pod "48124b9d-789a-4c58-9949-9a32c2f65854" (UID: "48124b9d-789a-4c58-9949-9a32c2f65854"). InnerVolumeSpecName "kube-api-access-4wk2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:03.321892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.321866    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:03.321892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.321888    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wk2z\" (UniqueName: \"kubernetes.io/projected/48124b9d-789a-4c58-9949-9a32c2f65854-kube-api-access-4wk2z\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:03.322066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.321899    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48124b9d-789a-4c58-9949-9a32c2f65854-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:03.944702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.944674    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"
Nov 03 19:37:03.944857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.944696    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2" event={"ID":"48124b9d-789a-4c58-9949-9a32c2f65854","Type":"ContainerDied","Data":"bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a"}
Nov 03 19:37:03.944857 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:03.944752    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb098ee9c3b47f0cedd52e884c68920ff21769300d9e96bb00b02288f2ec102a"
Nov 03 19:37:04.754920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.754896    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:37:04.837075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837040    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837090    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837141    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837179    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837237    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837282    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837322    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837354    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837406    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\") pod \"0cf7dd66-530e-41f3-abe3-d888e11e5526\" (UID: \"0cf7dd66-530e-41f3-abe3-d888e11e5526\") "
Nov 03 19:37:04.837855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.837813    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:04.838127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.838099    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:04.838260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.838225    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.838333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.838268    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf7dd66-530e-41f3-abe3-d888e11e5526-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.838333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.838284    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:04.838453 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.838352    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:04.841119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.840318    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:04.841470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.841414    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:04.841470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.841439    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "kube-api-access-xrh74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:04.841640 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.841523    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:37:04.841870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.841828    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token" pod "0cf7dd66-530e-41f3-abe3-d888e11e5526" (UID: "0cf7dd66-530e-41f3-abe3-d888e11e5526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:04.895426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.895403    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:37:04.938999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.938968    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.938999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939000    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.939227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939014    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.939227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939028    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.939227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939041    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.939227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939053    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf7dd66-530e-41f3-abe3-d888e11e5526-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.939227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.939064    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrh74\" (UniqueName: \"kubernetes.io/projected/0cf7dd66-530e-41f3-abe3-d888e11e5526-kube-api-access-xrh74\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:04.954120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.954084    4944 generic.go:356] "Generic (PLEG): container finished" podID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerID="7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08" exitCode=0
Nov 03 19:37:04.954290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.954142    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"
Nov 03 19:37:04.954290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.954183    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" event={"ID":"16d8d9e4-15e8-46da-9b31-39821a8988a2","Type":"ContainerDied","Data":"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"}
Nov 03 19:37:04.954290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.954243    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd" event={"ID":"16d8d9e4-15e8-46da-9b31-39821a8988a2","Type":"ContainerDied","Data":"36dbb1c6f2a8d5e34ac400fafec48b7cf167761594a9682965896e28f2574ebf"}
Nov 03 19:37:04.954290 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.954273    4944 scope.go:122] "RemoveContainer" containerID="7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"
Nov 03 19:37:04.956582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.956560    4944 generic.go:356] "Generic (PLEG): container finished" podID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerID="d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb" exitCode=0
Nov 03 19:37:04.956691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.956620    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"
Nov 03 19:37:04.956691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.956643    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerDied","Data":"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"}
Nov 03 19:37:04.956848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.956696    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp" event={"ID":"0cf7dd66-530e-41f3-abe3-d888e11e5526","Type":"ContainerDied","Data":"f33189b3204c7e1fb5792e27750cf487d47a5ce3c023488ef5cc6df553b7712d"}
Nov 03 19:37:04.961634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.961612    4944 generic.go:356] "Generic (PLEG): container finished" podID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerID="1f63272dfae63ad534fda8c1eefcf9039314828038a415a6a093b9e985c2552b" exitCode=0
Nov 03 19:37:04.961736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.961660    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" event={"ID":"dfd134d9-e71f-43e1-910a-d37a78a0a49f","Type":"ContainerDied","Data":"1f63272dfae63ad534fda8c1eefcf9039314828038a415a6a093b9e985c2552b"}
Nov 03 19:37:04.965818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.965772    4944 scope.go:122] "RemoveContainer" containerID="7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"
Nov 03 19:37:04.966160 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:04.966127    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08\": container with ID starting with 7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08 not found: ID does not exist" containerID="7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"
Nov 03 19:37:04.966368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.966171    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08"} err="failed to get container status \"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08\": rpc error: code = NotFound desc = could not find container \"7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08\": container with ID starting with 7e1443f32fc3cf33a4c61f55316783390be22ee7e19b91ee2388018a2f93ee08 not found: ID does not exist"
Nov 03 19:37:04.966368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.966362    4944 scope.go:122] "RemoveContainer" containerID="d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"
Nov 03 19:37:04.976098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.976076    4944 scope.go:122] "RemoveContainer" containerID="28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"
Nov 03 19:37:04.984875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.984853    4944 scope.go:122] "RemoveContainer" containerID="d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"
Nov 03 19:37:04.985187 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:04.985157    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb\": container with ID starting with d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb not found: ID does not exist" containerID="d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"
Nov 03 19:37:04.985278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.985190    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb"} err="failed to get container status \"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb\": rpc error: code = NotFound desc = could not find container \"d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb\": container with ID starting with d06441ed800d980034e529f1b27fb877c9cf40d2fefd274946f5aea369f8daeb not found: ID does not exist"
Nov 03 19:37:04.985278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.985213    4944 scope.go:122] "RemoveContainer" containerID="28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"
Nov 03 19:37:04.985462 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:04.985443    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409\": container with ID starting with 28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409 not found: ID does not exist" containerID="28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"
Nov 03 19:37:04.985501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:04.985468    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409"} err="failed to get container status \"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409\": rpc error: code = NotFound desc = could not find container \"28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409\": container with ID starting with 28fbf651316c643f88e87f70ef57de8f70d28f092b6a96330c619a0b33906409 not found: ID does not exist"
Nov 03 19:37:05.036093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.035937    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:37:05.039457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.039429    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\") pod \"16d8d9e4-15e8-46da-9b31-39821a8988a2\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") "
Nov 03 19:37:05.039603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.039548    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\" (UniqueName: \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\") pod \"16d8d9e4-15e8-46da-9b31-39821a8988a2\" (UID: \"16d8d9e4-15e8-46da-9b31-39821a8988a2\") "
Nov 03 19:37:05.039837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.039815    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume" pod "16d8d9e4-15e8-46da-9b31-39821a8988a2" (UID: "16d8d9e4-15e8-46da-9b31-39821a8988a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:05.039922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.039909    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16d8d9e4-15e8-46da-9b31-39821a8988a2-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:05.041541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.041517    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq" pod "16d8d9e4-15e8-46da-9b31-39821a8988a2" (UID: "16d8d9e4-15e8-46da-9b31-39821a8988a2"). InnerVolumeSpecName "kube-api-access-zphsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:05.140431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.140399    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\") pod \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") "
Nov 03 19:37:05.140431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.140436    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\" (UniqueName: \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\") pod \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\" (UID: \"dfd134d9-e71f-43e1-910a-d37a78a0a49f\") "
Nov 03 19:37:05.140755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.140731    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zphsq\" (UniqueName: \"kubernetes.io/projected/16d8d9e4-15e8-46da-9b31-39821a8988a2-kube-api-access-zphsq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:05.140891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.140757    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume" pod "dfd134d9-e71f-43e1-910a-d37a78a0a49f" (UID: "dfd134d9-e71f-43e1-910a-d37a78a0a49f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:05.142496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.142472    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d" pod "dfd134d9-e71f-43e1-910a-d37a78a0a49f" (UID: "dfd134d9-e71f-43e1-910a-d37a78a0a49f"). InnerVolumeSpecName "kube-api-access-z8w4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:05.176991 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.176965    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:37:05.180084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.180062    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fcb646-openshift-default-57fbd849bf-nsvpp"]
Nov 03 19:37:05.224203 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:05.224170    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:37:05.242071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.242050    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8w4d\" (UniqueName: \"kubernetes.io/projected/dfd134d9-e71f-43e1-910a-d37a78a0a49f-kube-api-access-z8w4d\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:05.242071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.242073    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd134d9-e71f-43e1-910a-d37a78a0a49f-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:05.347934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.347867    4944 scope.go:122] "RemoveContainer" containerID="6d21ff123e0fe8b6ce581d94f37661ee5f3dec53e09c6339dd8796de3292955e"
Nov 03 19:37:05.356736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.356706    4944 scope.go:122] "RemoveContainer" containerID="dc9ec7aff630ee8cb9beded49b46b31464911103d58577fe507cd90cbb9524aa"
Nov 03 19:37:05.367149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.367123    4944 scope.go:122] "RemoveContainer" containerID="1f63272dfae63ad534fda8c1eefcf9039314828038a415a6a093b9e985c2552b"
Nov 03 19:37:05.377329 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.377311    4944 scope.go:122] "RemoveContainer" containerID="233f1dd3d40e655b59aa5849508b1b6d7e16d07e4d0cc4fb5638d97d3f8f949e"
Nov 03 19:37:05.389358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.389331    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:37:05.395733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.395715    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fcb646-5c5b959bb6-vp2wd"]
Nov 03 19:37:05.933482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.933451    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" path="/var/lib/kubelet/pods/0cf7dd66-530e-41f3-abe3-d888e11e5526/volumes"
Nov 03 19:37:05.933875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.933867    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" path="/var/lib/kubelet/pods/16d8d9e4-15e8-46da-9b31-39821a8988a2/volumes"
Nov 03 19:37:05.966679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.966652    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm" event={"ID":"dfd134d9-e71f-43e1-910a-d37a78a0a49f","Type":"ContainerDied","Data":"fea310309567012333f64a2e7993cdb4d2562a67917f256209de02030ba4456b"}
Nov 03 19:37:05.966826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:05.966719    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"
Nov 03 19:37:06.178745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:06.178704    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:37:06.182486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:06.182456    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fcb646-78c47dfd4c-98fqm"]
Nov 03 19:37:06.819771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:06.819731    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:07.932312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:07.932277    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" path="/var/lib/kubelet/pods/dfd134d9-e71f-43e1-910a-d37a78a0a49f/volumes"
Nov 03 19:37:11.044388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.044355    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"]
Nov 03 19:37:11.090632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.090600    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8709ada9-qjwjp"]
Nov 03 19:37:11.371517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.371425    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:11.668106 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.668024    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:37:11.768588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.768548    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:37:11.933060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:11.932984    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e309f572-cbdb-4354-bcbb-048125161fda" path="/var/lib/kubelet/pods/e309f572-cbdb-4354-bcbb-048125161fda/volumes"
Nov 03 19:37:12.120846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:12.120783    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:12.373816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:12.373764    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:15.871168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:15.871135    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:16.668535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.668478    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" podUID="65979675-0926-4ce6-a17c-62db3d89992d" containerName="payload-processing" containerID="cri-o://ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c" gracePeriod=25
Nov 03 19:37:16.768968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.768930    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing" containerID="cri-o://115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974" gracePeriod=25
Nov 03 19:37:16.912352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.912323    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:37:16.960349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.960269    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\" (UniqueName: \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\") pod \"65979675-0926-4ce6-a17c-62db3d89992d\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") "
Nov 03 19:37:16.960349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.960325    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\") pod \"65979675-0926-4ce6-a17c-62db3d89992d\" (UID: \"65979675-0926-4ce6-a17c-62db3d89992d\") "
Nov 03 19:37:16.960749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.960722    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume" pod "65979675-0926-4ce6-a17c-62db3d89992d" (UID: "65979675-0926-4ce6-a17c-62db3d89992d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:16.962437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:16.962414    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn" pod "65979675-0926-4ce6-a17c-62db3d89992d" (UID: "65979675-0926-4ce6-a17c-62db3d89992d"). InnerVolumeSpecName "kube-api-access-nhbwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:17.026816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.026761    4944 generic.go:356] "Generic (PLEG): container finished" podID="65979675-0926-4ce6-a17c-62db3d89992d" containerID="ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c" exitCode=0
Nov 03 19:37:17.026989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.026834    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" event={"ID":"65979675-0926-4ce6-a17c-62db3d89992d","Type":"ContainerDied","Data":"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"}
Nov 03 19:37:17.026989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.026850    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"
Nov 03 19:37:17.026989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.026903    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb" event={"ID":"65979675-0926-4ce6-a17c-62db3d89992d","Type":"ContainerDied","Data":"54ecd94cc010e43b58b1726b35948a977571bbf1c5912d8e968e410ec944c6a0"}
Nov 03 19:37:17.026989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.026935    4944 scope.go:122] "RemoveContainer" containerID="ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"
Nov 03 19:37:17.038174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.038157    4944 scope.go:122] "RemoveContainer" containerID="ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"
Nov 03 19:37:17.038457 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:17.038438    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c\": container with ID starting with ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c not found: ID does not exist" containerID="ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"
Nov 03 19:37:17.038513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.038466    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c"} err="failed to get container status \"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c\": rpc error: code = NotFound desc = could not find container \"ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c\": container with ID starting with ec08cd0a891be663f46cc46adbec488a6eb838967c5abad0c030c836bfcc590c not found: ID does not exist"
Nov 03 19:37:17.061509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.061483    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhbwn\" (UniqueName: \"kubernetes.io/projected/65979675-0926-4ce6-a17c-62db3d89992d-kube-api-access-nhbwn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:17.061509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.061505    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65979675-0926-4ce6-a17c-62db3d89992d-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:17.063840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.063782    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"]
Nov 03 19:37:17.068737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.068708    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"]
Nov 03 19:37:17.070470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.070452    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a542e0a-rdlz9"]
Nov 03 19:37:17.072015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.071999    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30b86be4-r5gkj"]
Nov 03 19:37:17.179573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.179545    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:37:17.185561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.185536    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24cdad-6c65c687b6-xd9cb"]
Nov 03 19:37:17.933604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.933568    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65979675-0926-4ce6-a17c-62db3d89992d" path="/var/lib/kubelet/pods/65979675-0926-4ce6-a17c-62db3d89992d/volumes"
Nov 03 19:37:17.934023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.933995    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31" path="/var/lib/kubelet/pods/d08f3bb5-1384-4ddd-bf0b-7b9f371a9b31/volumes"
Nov 03 19:37:17.934351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:17.934337    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feae44a7-460c-4ed3-afe9-726bd8050d07" path="/var/lib/kubelet/pods/feae44a7-460c-4ed3-afe9-726bd8050d07/volumes"
Nov 03 19:37:19.856111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:19.856064    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.302526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.302491    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.552946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.552855    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.671054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.671021    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"]
Nov 03 19:37:21.716966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.716930    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-24cdad-4mjt2"]
Nov 03 19:37:21.796313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.796281    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:21.933127 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:21.933051    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48124b9d-789a-4c58-9949-9a32c2f65854" path="/var/lib/kubelet/pods/48124b9d-789a-4c58-9949-9a32c2f65854/volumes"
Nov 03 19:37:22.213580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:22.213486    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:25.044739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:25.044696    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:37:26.206433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:26.206394    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:28.653866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:28.653829    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:28.876312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:28.876274    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:37:28.876551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:28.876528    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy" containerID="cri-o://1572c36546352c277c7ac07831f1647850c3b63dff336274d39558806ccba94b" gracePeriod=30
Nov 03 19:37:28.904261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:28.904183    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.148774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.148736    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.468345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.468309    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:29.624184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624153    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"]
Nov 03 19:37:29.624547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624528    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:29.624547 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624541    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624558    4944 cpu_manager.go:414] "containerMap: removing container" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624564    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624576    4944 cpu_manager.go:414] "containerMap: removing container" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624581    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624595    4944 cpu_manager.go:414] "containerMap: removing container" podUID="65979675-0926-4ce6-a17c-62db3d89992d" containerName="payload-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624600    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65979675-0926-4ce6-a17c-62db3d89992d" containerName="payload-processing"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624609    4944 cpu_manager.go:414] "containerMap: removing container" podUID="48124b9d-789a-4c58-9949-9a32c2f65854" containerName="revoke-keys"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624614    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48124b9d-789a-4c58-9949-9a32c2f65854" containerName="revoke-keys"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624628    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:29.624641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624633    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:29.624987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624700    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:29.624987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624714    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd134d9-e71f-43e1-910a-d37a78a0a49f" containerName="payload-pre-processing"
Nov 03 19:37:29.624987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624731    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="16d8d9e4-15e8-46da-9b31-39821a8988a2" containerName="payload-processing"
Nov 03 19:37:29.624987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624741    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="65979675-0926-4ce6-a17c-62db3d89992d" containerName="payload-processing"
Nov 03 19:37:29.624987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.624747    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="48124b9d-789a-4c58-9949-9a32c2f65854" containerName="revoke-keys"
Nov 03 19:37:29.629394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.629377    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.679108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.679076    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:37:29.680159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.679863    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.680159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.679949    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.680159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.680054    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8zvn\" (UniqueName: \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.724887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.724809    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:37:29.724887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.724807    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:37:29.724887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.724848    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:37:29.726667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.726647    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"]
Nov 03 19:37:29.781483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.781454    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8zvn\" (UniqueName: \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.781623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.781510    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.781623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.781554    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.782209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.782188    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.783893 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.783870    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.838552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.838526    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8zvn\" (UniqueName: \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\") pod \"maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:29.941765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:29.941733    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:30.310740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:30.310663    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"]
Nov 03 19:37:30.316774 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:37:30.316741    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d9922d_068c_47d3_a8f7_fe90ace94bd8.slice/crio-234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462 WatchSource:0}: Error finding container 234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462: Status 404 returned error can't find the container with id 234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462
Nov 03 19:37:31.044853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:31.044821    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:31.104677 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:31.104650    4944 generic.go:356] "Generic (PLEG): container finished" podID="f4d9922d-068c-47d3-a8f7-fe90ace94bd8" containerID="a6c9574f0e0859964384affcfd10dedd336eb3a9ffc43130e11406c346b90c04" exitCode=0
Nov 03 19:37:31.104847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:31.104694    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw" event={"ID":"f4d9922d-068c-47d3-a8f7-fe90ace94bd8","Type":"ContainerDied","Data":"a6c9574f0e0859964384affcfd10dedd336eb3a9ffc43130e11406c346b90c04"}
Nov 03 19:37:31.104847 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:31.104739    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw" event={"ID":"f4d9922d-068c-47d3-a8f7-fe90ace94bd8","Type":"ContainerStarted","Data":"234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462"}
Nov 03 19:37:32.250580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.250555    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:32.308381 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.308353    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\") pod \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") "
Nov 03 19:37:32.308539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.308441    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\") pod \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") "
Nov 03 19:37:32.308539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.308485    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\" (UniqueName: \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\") pod \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\" (UID: \"f4d9922d-068c-47d3-a8f7-fe90ace94bd8\") "
Nov 03 19:37:32.308728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.308696    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca" pod "f4d9922d-068c-47d3-a8f7-fe90ace94bd8" (UID: "f4d9922d-068c-47d3-a8f7-fe90ace94bd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:32.310498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.310468    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca" pod "f4d9922d-068c-47d3-a8f7-fe90ace94bd8" (UID: "f4d9922d-068c-47d3-a8f7-fe90ace94bd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:32.310613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.310551    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn" pod "f4d9922d-068c-47d3-a8f7-fe90ace94bd8" (UID: "f4d9922d-068c-47d3-a8f7-fe90ace94bd8"). InnerVolumeSpecName "kube-api-access-d8zvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:32.410277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.410204    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:32.410277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.410231    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:32.410277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:32.410242    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8zvn\" (UniqueName: \"kubernetes.io/projected/f4d9922d-068c-47d3-a8f7-fe90ace94bd8-kube-api-access-d8zvn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:33.116081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:33.116046    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"
Nov 03 19:37:33.116270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:33.116075    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw" event={"ID":"f4d9922d-068c-47d3-a8f7-fe90ace94bd8","Type":"ContainerDied","Data":"234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462"}
Nov 03 19:37:33.116270 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:33.116139    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="234283cb04cd287fabd18ef1899c3d5540946da6a9edc73af990bcbcfb970462"
Nov 03 19:37:34.124455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.124427    4944 generic.go:356] "Generic (PLEG): container finished" podID="a23466c0-a35e-4646-b152-859d188fa066" containerID="1572c36546352c277c7ac07831f1647850c3b63dff336274d39558806ccba94b" exitCode=0
Nov 03 19:37:34.124889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.124495    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerDied","Data":"1572c36546352c277c7ac07831f1647850c3b63dff336274d39558806ccba94b"}
Nov 03 19:37:34.124889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.124552    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk" event={"ID":"a23466c0-a35e-4646-b152-859d188fa066","Type":"ContainerDied","Data":"87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c"}
Nov 03 19:37:34.124889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.124571    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ffadff1a83bd71564e76446e7bd7281520d1b2dcaab4127de84a1498f8b84c"
Nov 03 19:37:34.124889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.124584    4944 scope.go:122] "RemoveContainer" containerID="966b288281c61d6cb01a21e9b2576cd242ca2ed78c301507b8397879e1663811"
Nov 03 19:37:34.126934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.126916    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:37:34.225646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225571    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.225646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225638    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.225856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225819    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.225926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225913    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.225983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225941    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.226108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.225986    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.226108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226012    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.226108 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226038    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.226285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226095    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\") pod \"a23466c0-a35e-4646-b152-859d188fa066\" (UID: \"a23466c0-a35e-4646-b152-859d188fa066\") "
Nov 03 19:37:34.226285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226211    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:34.226285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226238    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:34.226469 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226387    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:34.226533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226464    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:34.226533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226516    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a23466c0-a35e-4646-b152-859d188fa066-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.226533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226531    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.226697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226543    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.226697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.226581    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:34.227914 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.227886    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "kube-api-access-m5m75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:34.228053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.228018    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:37:34.228426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.228411    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:34.228496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.228413    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token" pod "a23466c0-a35e-4646-b152-859d188fa066" (UID: "a23466c0-a35e-4646-b152-859d188fa066"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:34.327194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327156    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.327194 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327189    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a23466c0-a35e-4646-b152-859d188fa066-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.327417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327203    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.327417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327217    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.327417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327229    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a23466c0-a35e-4646-b152-859d188fa066-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.327417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.327241    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5m75\" (UniqueName: \"kubernetes.io/projected/a23466c0-a35e-4646-b152-859d188fa066-kube-api-access-m5m75\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:34.335195 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.335162    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:37:34.336760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.336736    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:37:34.514615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.514580    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:34.972660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:34.972549    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:35.045044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.045004    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:37:35.131331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.131298    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"
Nov 03 19:37:35.224100 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:35.224022    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:37:35.348628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.348594    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:37:35.352662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.352635    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-24cdad-openshift-default-54b5f9d7bf-jhstk"]
Nov 03 19:37:35.883997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.883967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:35.932277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:35.932245    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a23466c0-a35e-4646-b152-859d188fa066" path="/var/lib/kubelet/pods/a23466c0-a35e-4646-b152-859d188fa066/volumes"
Nov 03 19:37:37.014761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.014737    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:37:37.143519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.143438    4944 generic.go:356] "Generic (PLEG): container finished" podID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerID="115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974" exitCode=0
Nov 03 19:37:37.143519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.143487    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"
Nov 03 19:37:37.143733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.143518    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" event={"ID":"f2a780f3-f7ae-4499-ba3e-ff4e7802d425","Type":"ContainerDied","Data":"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"}
Nov 03 19:37:37.143733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.143573    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs" event={"ID":"f2a780f3-f7ae-4499-ba3e-ff4e7802d425","Type":"ContainerDied","Data":"83a25fe98d7c01bc87e6a8697edaf62febd7a3a83c8e33239faf088330c41c3d"}
Nov 03 19:37:37.143733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.143598    4944 scope.go:122] "RemoveContainer" containerID="115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"
Nov 03 19:37:37.152327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.152309    4944 scope.go:122] "RemoveContainer" containerID="115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"
Nov 03 19:37:37.152477 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.152460    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\") pod \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") "
Nov 03 19:37:37.152611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.152598    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\" (UniqueName: \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\") pod \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\" (UID: \"f2a780f3-f7ae-4499-ba3e-ff4e7802d425\") "
Nov 03 19:37:37.152611 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:37.152600    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974\": container with ID starting with 115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974 not found: ID does not exist" containerID="115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"
Nov 03 19:37:37.152697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.152628    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974"} err="failed to get container status \"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974\": rpc error: code = NotFound desc = could not find container \"115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974\": container with ID starting with 115312ebdf5fc836bf75a48f5365f8f7a7e91f58c8568970783512a2085d4974 not found: ID does not exist"
Nov 03 19:37:37.152872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.152850    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume" pod "f2a780f3-f7ae-4499-ba3e-ff4e7802d425" (UID: "f2a780f3-f7ae-4499-ba3e-ff4e7802d425"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:37.154584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.154563    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h" pod "f2a780f3-f7ae-4499-ba3e-ff4e7802d425" (UID: "f2a780f3-f7ae-4499-ba3e-ff4e7802d425"). InnerVolumeSpecName "kube-api-access-8sv8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:37.253634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.253603    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv8h\" (UniqueName: \"kubernetes.io/projected/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-kube-api-access-8sv8h\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:37.253634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.253632    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a780f3-f7ae-4499-ba3e-ff4e7802d425-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:37.555652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.555620    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:37:37.600800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.600759    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24cdad-658f87bcc5-7kvfs"]
Nov 03 19:37:37.939943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:37.939909    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" path="/var/lib/kubelet/pods/f2a780f3-f7ae-4499-ba3e-ff4e7802d425/volumes"
Nov 03 19:37:41.350080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:41.350043    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 19:37:42.394115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:42.394082    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:42.644959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:42.644880    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:44.992193 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:44.992160    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:37:46.350642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:46.350583    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing" containerID="cri-o://c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe" gracePeriod=25
Nov 03 19:37:48.901348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901310    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"]
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901680    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901691    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901708    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing"
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901714    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing"
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901723    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f4d9922d-068c-47d3-a8f7-fe90ace94bd8" containerName="revoke-keys"
Nov 03 19:37:48.901750 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901729    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d9922d-068c-47d3-a8f7-fe90ace94bd8" containerName="revoke-keys"
Nov 03 19:37:48.901981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901819    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:37:48.901981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901833    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2a780f3-f7ae-4499-ba3e-ff4e7802d425" containerName="payload-pre-processing"
Nov 03 19:37:48.901981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901843    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d9922d-068c-47d3-a8f7-fe90ace94bd8" containerName="revoke-keys"
Nov 03 19:37:48.901981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901851    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf7dd66-530e-41f3-abe3-d888e11e5526" containerName="istio-proxy"
Nov 03 19:37:48.901981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.901860    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:37:48.906384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.906368    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:48.955459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.955432    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:37:48.958102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:48.958081    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:37:49.004535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.004506    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:37:49.004535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.004521    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:37:49.006376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.006352    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"]
Nov 03 19:37:49.064384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.064345    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h42bz\" (UniqueName: \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.064567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.064487    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.064567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.064541    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.166317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.166236    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h42bz\" (UniqueName: \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.166317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.166286    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.166554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.166327    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.167052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.167027    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.168636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.168618    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.223639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.223590    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h42bz\" (UniqueName: \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\") pod \"maas-api-revoke-sub-61cef7d1-75rkn\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.355156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.355118    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main" containerID="cri-o://37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6" gracePeriod=45
Nov 03 19:37:49.355864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.355517    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main" containerID="cri-o://eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5" gracePeriod=45
Nov 03 19:37:49.519441 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.519408    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:49.618709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.618681    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:37:49.620785 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.620765    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:37:49.772135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772104    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772207    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772237    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772287    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772328    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772355    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772399    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\" (UniqueName: \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772409    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772426    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\" (UniqueName: \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772470    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772512    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772544    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772564    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\") pod \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\" (UID: \"9a9b4d29-2199-48e3-924c-f0ea100ac0c0\") "
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772597    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\" (UniqueName: \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\") pod \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\" (UID: \"15e1f75f-cf6e-44f6-b08c-c2eb20760e00\") "
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772591    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772652    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.772903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772839    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772979    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772997    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.773009    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.773023    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.773048    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.773410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.772987    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.774816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.774768    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.775145 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.775111    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:37:49.775277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.775177    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:49.775347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.775311    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78" pod "15e1f75f-cf6e-44f6-b08c-c2eb20760e00" (UID: "15e1f75f-cf6e-44f6-b08c-c2eb20760e00"). InnerVolumeSpecName "kube-api-access-2lp78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:49.775426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.775357    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:49.775489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.775440    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg" pod "9a9b4d29-2199-48e3-924c-f0ea100ac0c0" (UID: "9a9b4d29-2199-48e3-924c-f0ea100ac0c0"). InnerVolumeSpecName "kube-api-access-zbjsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:49.873541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873508    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873544    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873560    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjsg\" (UniqueName: \"kubernetes.io/projected/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-kube-api-access-zbjsg\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873570    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873579    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a9b4d29-2199-48e3-924c-f0ea100ac0c0-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873593    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lp78\" (UniqueName: \"kubernetes.io/projected/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-kube-api-access-2lp78\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.873756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.873604    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15e1f75f-cf6e-44f6-b08c-c2eb20760e00-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:49.951864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:49.951834    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"]
Nov 03 19:37:49.958769 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:37:49.958738    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cc9e66_3441_46b4_9c8b_1fbed3215a8b.slice/crio-75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f WatchSource:0}: Error finding container 75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f: Status 404 returned error can't find the container with id 75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f
Nov 03 19:37:50.222331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.222301    4944 generic.go:356] "Generic (PLEG): container finished" podID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerID="eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5" exitCode=0
Nov 03 19:37:50.222445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.222333    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" event={"ID":"15e1f75f-cf6e-44f6-b08c-c2eb20760e00","Type":"ContainerDied","Data":"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"}
Nov 03 19:37:50.222445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.222380    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l" event={"ID":"15e1f75f-cf6e-44f6-b08c-c2eb20760e00","Type":"ContainerDied","Data":"6341a0c370061ba63e932bca164194230281e2c1d5a56653d4b0e653bd7a3429"}
Nov 03 19:37:50.222445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.222345    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"
Nov 03 19:37:50.222445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.222398    4944 scope.go:122] "RemoveContainer" containerID="eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"
Nov 03 19:37:50.224105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.224084    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn" event={"ID":"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b","Type":"ContainerStarted","Data":"75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f"}
Nov 03 19:37:50.225493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.225474    4944 generic.go:356] "Generic (PLEG): container finished" podID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerID="37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6" exitCode=0
Nov 03 19:37:50.225582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.225506    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" event={"ID":"9a9b4d29-2199-48e3-924c-f0ea100ac0c0","Type":"ContainerDied","Data":"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"}
Nov 03 19:37:50.225582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.225531    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"
Nov 03 19:37:50.225658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.225537    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr" event={"ID":"9a9b4d29-2199-48e3-924c-f0ea100ac0c0","Type":"ContainerDied","Data":"7c5e0d35ece73d6e4f51acebe2704f14712d4a6b52e6ee2c0da5dac883bad9d3"}
Nov 03 19:37:50.231621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.231606    4944 scope.go:122] "RemoveContainer" containerID="eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"
Nov 03 19:37:50.231887 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:50.231872    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5\": container with ID starting with eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5 not found: ID does not exist" containerID="eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"
Nov 03 19:37:50.231954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.231895    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5"} err="failed to get container status \"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5\": rpc error: code = NotFound desc = could not find container \"eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5\": container with ID starting with eaac01f14f2185e246c299e8cb31ee6757b05834c89e8b76e0f775ddf3d4daf5 not found: ID does not exist"
Nov 03 19:37:50.231954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.231911    4944 scope.go:122] "RemoveContainer" containerID="37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"
Nov 03 19:37:50.239516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.239501    4944 scope.go:122] "RemoveContainer" containerID="37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"
Nov 03 19:37:50.239770 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:37:50.239751    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6\": container with ID starting with 37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6 not found: ID does not exist" containerID="37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"
Nov 03 19:37:50.239842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.239779    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6"} err="failed to get container status \"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6\": rpc error: code = NotFound desc = could not find container \"37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6\": container with ID starting with 37da473def9e9855eb5b4a23e72a6956166a4f5b6eb13d126c55f79f343519f6 not found: ID does not exist"
Nov 03 19:37:50.375058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.375029    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:37:50.379292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.379233    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-619731/distinct-w2-619731-kserve-bd9668bf4-vkphr"]
Nov 03 19:37:50.480229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.480195    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:37:50.484125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:50.484100    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-619731/premium-w2-619731-kserve-87dfc47f5-lss4l"]
Nov 03 19:37:51.153916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:51.153822    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:37:51.232613 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:51.232573    4944 generic.go:356] "Generic (PLEG): container finished" podID="f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" containerID="d274fef006b8e3ebcb15afa4c9a2df2e0e502ea7c2400e6f107a3ba012823991" exitCode=0
Nov 03 19:37:51.232848 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:51.232646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn" event={"ID":"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b","Type":"ContainerDied","Data":"d274fef006b8e3ebcb15afa4c9a2df2e0e502ea7c2400e6f107a3ba012823991"}
Nov 03 19:37:51.934660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:51.934624    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" path="/var/lib/kubelet/pods/15e1f75f-cf6e-44f6-b08c-c2eb20760e00/volumes"
Nov 03 19:37:51.935275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:51.935255    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" path="/var/lib/kubelet/pods/9a9b4d29-2199-48e3-924c-f0ea100ac0c0/volumes"
Nov 03 19:37:52.370887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.370863    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:52.498646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.498570    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\") pod \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") "
Nov 03 19:37:52.498646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.498613    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\") pod \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") "
Nov 03 19:37:52.498925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.498676    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\" (UniqueName: \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\") pod \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\" (UID: \"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b\") "
Nov 03 19:37:52.499116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.499088    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca" pod "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" (UID: "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:37:52.500833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.500805    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca" pod "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" (UID: "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:37:52.500979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.500901    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz" pod "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" (UID: "f8cc9e66-3441-46b4-9c8b-1fbed3215a8b"). InnerVolumeSpecName "kube-api-access-h42bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:37:52.599680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.599641    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:52.599680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.599674    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:52.599680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:52.599684    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h42bz\" (UniqueName: \"kubernetes.io/projected/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b-kube-api-access-h42bz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:37:53.247240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:53.247208    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"
Nov 03 19:37:53.247415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:53.247207    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn" event={"ID":"f8cc9e66-3441-46b4-9c8b-1fbed3215a8b","Type":"ContainerDied","Data":"75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f"}
Nov 03 19:37:53.247415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:37:53.247334    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75088ed664a31d158d168018d14ccb390909ebe21e1f8db9f558b97bdb32297f"
Nov 03 19:38:01.154140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:01.154101    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:38:04.463732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:04.463695    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:05.225398 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:38:05.225351    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:38:05.424757 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:05.424712    4944 scope.go:122] "RemoveContainer" containerID="943ee3792450ade441da6bf8ae83ef2165d8ee8a92281dd78616a94ea07ccb03"
Nov 03 19:38:05.436070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:05.436046    4944 scope.go:122] "RemoveContainer" containerID="1cba25435172b39d6314a8fc4e1fbdef1199f1d29634b2ca31e61b294370b70e"
Nov 03 19:38:05.446197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:05.446174    4944 scope.go:122] "RemoveContainer" containerID="41ab8a7813ad722ed3cfef6ed6cd61cb75e86c91b49a7a15b6136b8d5cf1e388"
Nov 03 19:38:05.964430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:05.964395    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.248482 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.248444    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:06.333549 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.333506    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"]
Nov 03 19:38:06.379688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.379658    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-619731-p9mzw"]
Nov 03 19:38:06.597109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.597085    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 19:38:06.618443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.618418    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\") pod \"adbb7582-6d86-418a-8f25-c0af33550947\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") "
Nov 03 19:38:06.618596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.618520    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\" (UniqueName: \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\") pod \"adbb7582-6d86-418a-8f25-c0af33550947\" (UID: \"adbb7582-6d86-418a-8f25-c0af33550947\") "
Nov 03 19:38:06.618861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.618834    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume" pod "adbb7582-6d86-418a-8f25-c0af33550947" (UID: "adbb7582-6d86-418a-8f25-c0af33550947"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:06.620557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.620537    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96" pod "adbb7582-6d86-418a-8f25-c0af33550947" (UID: "adbb7582-6d86-418a-8f25-c0af33550947"). InnerVolumeSpecName "kube-api-access-srd96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:06.719249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.719216    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srd96\" (UniqueName: \"kubernetes.io/projected/adbb7582-6d86-418a-8f25-c0af33550947-kube-api-access-srd96\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:06.719249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.719243    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adbb7582-6d86-418a-8f25-c0af33550947-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:06.871231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:06.871142    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:07.321659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.321620    4944 generic.go:356] "Generic (PLEG): container finished" podID="adbb7582-6d86-418a-8f25-c0af33550947" containerID="c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe" exitCode=0
Nov 03 19:38:07.322123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.321678    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"
Nov 03 19:38:07.322123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.321708    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" event={"ID":"adbb7582-6d86-418a-8f25-c0af33550947","Type":"ContainerDied","Data":"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"}
Nov 03 19:38:07.322123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.321777    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4" event={"ID":"adbb7582-6d86-418a-8f25-c0af33550947","Type":"ContainerDied","Data":"2b10c3eb525bcd753216a0b0407b3b95282f458e8ddf1873a7c036e6e68faad3"}
Nov 03 19:38:07.322123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.321824    4944 scope.go:122] "RemoveContainer" containerID="c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"
Nov 03 19:38:07.331761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.331743    4944 scope.go:122] "RemoveContainer" containerID="c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"
Nov 03 19:38:07.332064 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:38:07.332036    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe\": container with ID starting with c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe not found: ID does not exist" containerID="c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"
Nov 03 19:38:07.332137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.332077    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe"} err="failed to get container status \"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe\": rpc error: code = NotFound desc = could not find container \"c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe\": container with ID starting with c831aea66d57cd87ed50d3e3231b5179ed2ed1eda6cafa211a3f8ab3f57da7fe not found: ID does not exist"
Nov 03 19:38:07.476977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.476938    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 19:38:07.523420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.523385    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-619731-574b8f8b59-t6ct4"]
Nov 03 19:38:07.932575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.932539    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbb7582-6d86-418a-8f25-c0af33550947" path="/var/lib/kubelet/pods/adbb7582-6d86-418a-8f25-c0af33550947/volumes"
Nov 03 19:38:07.932980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:07.932964    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d9922d-068c-47d3-a8f7-fe90ace94bd8" path="/var/lib/kubelet/pods/f4d9922d-068c-47d3-a8f7-fe90ace94bd8/volumes"
Nov 03 19:38:10.797005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:10.796971    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:11.047483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:11.047372    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:11.922823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:11.922775    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:12.176077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:12.175993    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:12.425912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:12.425876    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.389682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:13.389645    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.643849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:13.643756    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 19:38:13.644015 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:13.643986    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy" containerID="cri-o://e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38" gracePeriod=30
Nov 03 19:38:13.731391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:13.731364    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:13.977275 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:13.977200    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.413890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.413854    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:18.881529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.881502    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 19:38:18.928246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928184    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928246 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928221    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928254    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928428    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928471    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928521    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928539    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:18.928568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928562    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928630    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928656    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\") pod \"a822a620-0a77-49b0-b08f-ae0a750b90d9\" (UID: \"a822a620-0a77-49b0-b08f-ae0a750b90d9\") "
Nov 03 19:38:18.928833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928672    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:18.928833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928614    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928805    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.928882    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.929008    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.929026    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a822a620-0a77-49b0-b08f-ae0a750b90d9-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.929036    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.929045    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:18.929062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.929059    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:18.930737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.930708    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:38:18.930856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.930760    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:38:18.931024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.931009    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:18.931061 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:18.931030    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z" pod "a822a620-0a77-49b0-b08f-ae0a750b90d9" (UID: "a822a620-0a77-49b0-b08f-ae0a750b90d9"). InnerVolumeSpecName "kube-api-access-rl88z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:19.029579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.029547    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl88z\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-kube-api-access-rl88z\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:19.029579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.029572    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:19.029579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.029584    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:19.029831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.029592    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a822a620-0a77-49b0-b08f-ae0a750b90d9-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:19.387036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.387000    4944 generic.go:356] "Generic (PLEG): container finished" podID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerID="e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38" exitCode=0
Nov 03 19:38:19.387212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.387056    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"
Nov 03 19:38:19.387212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.387078    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerDied","Data":"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"}
Nov 03 19:38:19.387212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.387134    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh" event={"ID":"a822a620-0a77-49b0-b08f-ae0a750b90d9","Type":"ContainerDied","Data":"4c11ea6ea19ec0e815d511048dac3128fcb944eac30e38e639b33c3ad6db8a22"}
Nov 03 19:38:19.387212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.387155    4944 scope.go:122] "RemoveContainer" containerID="e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"
Nov 03 19:38:19.397183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.397167    4944 scope.go:122] "RemoveContainer" containerID="893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"
Nov 03 19:38:19.405401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.405381    4944 scope.go:122] "RemoveContainer" containerID="e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"
Nov 03 19:38:19.408566 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:38:19.408535    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38\": container with ID starting with e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38 not found: ID does not exist" containerID="e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"
Nov 03 19:38:19.408665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.408576    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38"} err="failed to get container status \"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38\": rpc error: code = NotFound desc = could not find container \"e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38\": container with ID starting with e99bf5a6ce96bf01c8489ea905b37d569de24b3d7bad4415698936c1b522fc38 not found: ID does not exist"
Nov 03 19:38:19.408665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.408600    4944 scope.go:122] "RemoveContainer" containerID="893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"
Nov 03 19:38:19.408926 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:38:19.408901    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942\": container with ID starting with 893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942 not found: ID does not exist" containerID="893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"
Nov 03 19:38:19.408998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.408940    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942"} err="failed to get container status \"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942\": rpc error: code = NotFound desc = could not find container \"893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942\": container with ID starting with 893e928de94a25f027cb6ee153da982377e2fe0edc6d2a0c32d426978a795942 not found: ID does not exist"
Nov 03 19:38:19.541013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.540972    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 19:38:19.586940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.586910    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-619731-openshift-default-57d8bd644c-bxsrh"]
Nov 03 19:38:19.932225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:19.932195    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" path="/var/lib/kubelet/pods/a822a620-0a77-49b0-b08f-ae0a750b90d9/volumes"
Nov 03 19:38:35.224838 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:38:35.224782    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:38:35.354209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:35.354172    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:35.600603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:35.600569    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:35.842948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:35.842916    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:42.358881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.358843    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"]
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359196    4944 cpu_manager.go:414] "containerMap: removing container" podUID="f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" containerName="revoke-keys"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359207    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" containerName="revoke-keys"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359222    4944 cpu_manager.go:414] "containerMap: removing container" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359228    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359236    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359241    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359257    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359262    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359271    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359276    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a23466c0-a35e-4646-b152-859d188fa066" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359284    4944 cpu_manager.go:414] "containerMap: removing container" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359289    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359295    4944 cpu_manager.go:414] "containerMap: removing container" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359299    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359358    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359367    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbb7582-6d86-418a-8f25-c0af33550947" containerName="payload-pre-processing"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359373    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a9b4d29-2199-48e3-924c-f0ea100ac0c0" containerName="main"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359382    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" containerName="revoke-keys"
Nov 03 19:38:42.359380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.359388    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e1f75f-cf6e-44f6-b08c-c2eb20760e00" containerName="main"
Nov 03 19:38:42.362479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.362463    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.411264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.411235    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:38:42.411428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.411352    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:38:42.411428 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.411400    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:38:42.459327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.459299    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"]
Nov 03 19:38:42.511532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.511506    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:38:42.541017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.540987    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6fh\" (UniqueName: \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.541175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.541025    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.541175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.541105    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.641626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.641555    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6fh\" (UniqueName: \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.641626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.641593    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.641825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.641627    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.642271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.642250    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.643859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.643839    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.700510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.700477    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6fh\" (UniqueName: \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\") pod \"maas-api-revoke-sub-85237e45-hjwlr\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:42.975772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:42.975691    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:43.342409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:43.342374    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"]
Nov 03 19:38:43.349116 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:38:43.349091    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07180f7_868c_4d70_8ff4_c2671439c540.slice/crio-80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8 WatchSource:0}: Error finding container 80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8: Status 404 returned error can't find the container with id 80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8
Nov 03 19:38:43.507698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:43.507659    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr" event={"ID":"a07180f7-868c-4d70-8ff4-c2671439c540","Type":"ContainerStarted","Data":"80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8"}
Nov 03 19:38:44.184341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.184308    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"]
Nov 03 19:38:44.184835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.184817    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a822a620-0a77-49b0-b08f-ae0a750b90d9" containerName="istio-proxy"
Nov 03 19:38:44.188046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.188022    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.236526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.236501    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 19:38:44.282702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.282669    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-26cvs\""
Nov 03 19:38:44.286172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.286138    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"]
Nov 03 19:38:44.357326 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357289    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357404    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357441    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7kd4\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-kube-api-access-p7kd4\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357490    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4623f-6a4c-4ed0-bad9-7883737d362a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357536    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357558    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357577    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357609    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.357692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.357677    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458247    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458289    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7kd4\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-kube-api-access-p7kd4\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458313    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4623f-6a4c-4ed0-bad9-7883737d362a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458337    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458356    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458503    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458570    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458602    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458636    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458739    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.458966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458856    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.459036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.458970    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.459081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.459055    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.459141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.459124    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19b4623f-6a4c-4ed0-bad9-7883737d362a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.460840 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.460816    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.461026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.461007    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.513980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.513951    4944 generic.go:356] "Generic (PLEG): container finished" podID="a07180f7-868c-4d70-8ff4-c2671439c540" containerID="3eb2e88b0e4da9fb462c1f45062f7d419d051af6c518028ed39342833e081f5b" exitCode=0
Nov 03 19:38:44.514412 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.514036    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr" event={"ID":"a07180f7-868c-4d70-8ff4-c2671439c540","Type":"ContainerDied","Data":"3eb2e88b0e4da9fb462c1f45062f7d419d051af6c518028ed39342833e081f5b"}
Nov 03 19:38:44.515522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.515496    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.515629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.515618    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7kd4\" (UniqueName: \"kubernetes.io/projected/19b4623f-6a4c-4ed0-bad9-7883737d362a-kube-api-access-p7kd4\") pod \"maas-default-gateway-openshift-default-77f556d647-stw9w\" (UID: \"19b4623f-6a4c-4ed0-bad9-7883737d362a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:44.800778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:44.800742    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:45.223350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.223270    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"]
Nov 03 19:38:45.229576 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:38:45.229549    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b4623f_6a4c_4ed0_bad9_7883737d362a.slice/crio-ea08ea23ce430ff22ec5a9eae6c182ccc67c8ba69043fec26f0d818eb0a44770 WatchSource:0}: Error finding container ea08ea23ce430ff22ec5a9eae6c182ccc67c8ba69043fec26f0d818eb0a44770: Status 404 returned error can't find the container with id ea08ea23ce430ff22ec5a9eae6c182ccc67c8ba69043fec26f0d818eb0a44770
Nov 03 19:38:45.231979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.231941    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:38:45.232075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.232027    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:38:45.232120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.232071    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:38:45.521059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.521018    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" event={"ID":"19b4623f-6a4c-4ed0-bad9-7883737d362a","Type":"ContainerStarted","Data":"64caf51ef7c5373b13bd5692cc3a3e8a8c62db4b1d75fbf38667a99a3255c258"}
Nov 03 19:38:45.521558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.521113    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" event={"ID":"19b4623f-6a4c-4ed0-bad9-7883737d362a","Type":"ContainerStarted","Data":"ea08ea23ce430ff22ec5a9eae6c182ccc67c8ba69043fec26f0d818eb0a44770"}
Nov 03 19:38:45.636240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.636178    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podStartSLOduration=1.6361586030000002 podStartE2EDuration="1.636158603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:38:45.633874817 +0000 UTC m=+3223.911228434" watchObservedRunningTime="2026-11-03 19:38:45.636158603 +0000 UTC m=+3223.913512187"
Nov 03 19:38:45.644873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.644846    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:45.670419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.670375    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\") pod \"a07180f7-868c-4d70-8ff4-c2671439c540\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") "
Nov 03 19:38:45.670544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.670463    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\" (UniqueName: \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\") pod \"a07180f7-868c-4d70-8ff4-c2671439c540\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") "
Nov 03 19:38:45.670544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.670508    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\") pod \"a07180f7-868c-4d70-8ff4-c2671439c540\" (UID: \"a07180f7-868c-4d70-8ff4-c2671439c540\") "
Nov 03 19:38:45.670892 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.670867    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca" pod "a07180f7-868c-4d70-8ff4-c2671439c540" (UID: "a07180f7-868c-4d70-8ff4-c2671439c540"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:38:45.672481 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.672456    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca" pod "a07180f7-868c-4d70-8ff4-c2671439c540" (UID: "a07180f7-868c-4d70-8ff4-c2671439c540"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:38:45.672573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.672558    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh" pod "a07180f7-868c-4d70-8ff4-c2671439c540" (UID: "a07180f7-868c-4d70-8ff4-c2671439c540"). InnerVolumeSpecName "kube-api-access-7f6fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:38:45.772022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.771924    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:45.772250 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.772229    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f6fh\" (UniqueName: \"kubernetes.io/projected/a07180f7-868c-4d70-8ff4-c2671439c540-kube-api-access-7f6fh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:45.772355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.772341    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a07180f7-868c-4d70-8ff4-c2671439c540-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:38:45.800844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:45.800812    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:46.542679 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:46.542650    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"
Nov 03 19:38:46.543138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:46.542683    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr" event={"ID":"a07180f7-868c-4d70-8ff4-c2671439c540","Type":"ContainerDied","Data":"80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8"}
Nov 03 19:38:46.543138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:46.542728    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ad57586a880667dec37d12059465c41fd0319c246dd27a8d957610879492e8"
Nov 03 19:38:46.803205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:46.803125    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:46.803205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:46.803175    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:47.802057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:47.802022    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:47.802443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:47.802074    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:48.801941 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:48.801905    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:48.802107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:48.801961    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:49.801661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:49.801627    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:49.801869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:49.801679    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:50.801126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:50.801087    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:50.801582 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:50.801150    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:51.801553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:51.801517    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:51.801994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:51.801567    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:52.801896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:52.801855    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused" start-of-body=
Nov 03 19:38:52.802296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:52.801932    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": dial tcp 10.134.1.12:15021: connect: connection refused"
Nov 03 19:38:53.804926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:53.804895    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:53.805389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:53.805206    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:53.805877 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:53.805861    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w"
Nov 03 19:38:54.341117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:54.341077    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:56.563928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:56.563895    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:38:59.301519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.301484    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"]
Nov 03 19:38:59.301924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.301891    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a07180f7-868c-4d70-8ff4-c2671439c540" containerName="revoke-keys"
Nov 03 19:38:59.301924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.301905    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a07180f7-868c-4d70-8ff4-c2671439c540" containerName="revoke-keys"
Nov 03 19:38:59.302046 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.301964    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a07180f7-868c-4d70-8ff4-c2671439c540" containerName="revoke-keys"
Nov 03 19:38:59.305472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.305452    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.402168 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.402130    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"]
Nov 03 19:38:59.415433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.415403    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.415620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.415540    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.415821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.415773    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94260b7-d303-41a2-b800-ce732a0b3a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.415961 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.415921    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.416035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.415978    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92st4\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-kube-api-access-92st4\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.416035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.416013    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.416177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.416045    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.416177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.416108    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.416177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.416170    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517529    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517578    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94260b7-d303-41a2-b800-ce732a0b3a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517688    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517761    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92st4\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-kube-api-access-92st4\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517812    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.517895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.517848    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518033    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518047    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518115    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518176    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518401    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94260b7-d303-41a2-b800-ce732a0b3a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518539    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518577    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.518649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.518588    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.520238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.520213    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.520688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.520666    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.575129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.575069    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92st4\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-kube-api-access-92st4\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.575227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.575211    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94260b7-d303-41a2-b800-ce732a0b3a8b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-h6d6n\" (UID: \"c94260b7-d303-41a2-b800-ce732a0b3a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:38:59.618340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:38:59.618315    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:39:00.039667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.039623    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"]
Nov 03 19:39:00.082624 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:39:00.082595    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94260b7_d303_41a2_b800_ce732a0b3a8b.slice/crio-98427f16e13d5f722ad721cd3bd47bfba6d6ff0d7fa0a9fa2124f8fa7dbfffc8 WatchSource:0}: Error finding container 98427f16e13d5f722ad721cd3bd47bfba6d6ff0d7fa0a9fa2124f8fa7dbfffc8: Status 404 returned error can't find the container with id 98427f16e13d5f722ad721cd3bd47bfba6d6ff0d7fa0a9fa2124f8fa7dbfffc8
Nov 03 19:39:00.084822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.084773    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:00.084937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.084895    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:00.085005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.084944    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:39:00.634998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.634948    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" event={"ID":"c94260b7-d303-41a2-b800-ce732a0b3a8b","Type":"ContainerStarted","Data":"237fbd7ddc6cac65969d7ecdb4034b0aef8ec68fb502ff49dcfb32e53ecbd6b8"}
Nov 03 19:39:00.635433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.635017    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" event={"ID":"c94260b7-d303-41a2-b800-ce732a0b3a8b","Type":"ContainerStarted","Data":"98427f16e13d5f722ad721cd3bd47bfba6d6ff0d7fa0a9fa2124f8fa7dbfffc8"}
Nov 03 19:39:00.748663 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:00.748607    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podStartSLOduration=1.748594661 podStartE2EDuration="1.748594661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:00.748335275 +0000 UTC m=+3239.025688860" watchObservedRunningTime="2026-11-03 19:39:00.748594661 +0000 UTC m=+3239.025948244"
Nov 03 19:39:01.621610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:01.621573    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:39:01.621931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:01.621899    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:01.622172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:01.622146    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:02.625708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:02.625244    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:02.625708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:02.625308    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:03.625048 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:03.625010    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:03.625333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:03.625303    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:04.626815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:04.626154    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:04.626815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:04.626227    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:05.243140 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:39:05.243102    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:39:05.495333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.495253    4944 scope.go:122] "RemoveContainer" containerID="b0049d8a334877b8e31538081d9eb29f9fdc0d2e9f8f65d6dbedcade6cc7d602"
Nov 03 19:39:05.503661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.503642    4944 scope.go:122] "RemoveContainer" containerID="48a5f24c8938ba11b35660077356a06e5a1e556135aba732915d353c6fe3070c"
Nov 03 19:39:05.536586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.536563    4944 scope.go:122] "RemoveContainer" containerID="939e609916ed0649392837b5a5def8170228b5deaa4e999c44c0d0987fe4def0"
Nov 03 19:39:05.545539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.545516    4944 scope.go:122] "RemoveContainer" containerID="44ee90ecbc629608caab63f25cc9f6737c0ecb9cb57e7c637c59a33acaf578f2"
Nov 03 19:39:05.553189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.553164    4944 scope.go:122] "RemoveContainer" containerID="c3e47056a8e9ebcb6cf16205d85170c6ec0be8a62397f5ba8503b635395ced90"
Nov 03 19:39:05.619535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.619113    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:05.619535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:05.619180    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:06.619553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:06.619519    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-h6d6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body=
Nov 03 19:39:06.619972 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:06.619582    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n" podUID="c94260b7-d303-41a2-b800-ce732a0b3a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused"
Nov 03 19:39:08.394394 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:08.394363    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:39:08.394920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:08.394845    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:39:08.395717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:08.395693    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-h6d6n"
Nov 03 19:39:19.054489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:19.054455    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"]
Nov 03 19:39:19.102056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:19.102022    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-997f7d6d-pk8pt"]
Nov 03 19:39:19.932511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:19.932475    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5e1be5-fd9e-4280-8790-076ff15b0c05" path="/var/lib/kubelet/pods/5f5e1be5-fd9e-4280-8790-076ff15b0c05/volumes"
Nov 03 19:39:29.045647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.045610    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"]
Nov 03 19:39:29.092236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.092204    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"]
Nov 03 19:39:29.093762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.093742    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-770a030d-c2cw9"]
Nov 03 19:39:29.095089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.095067    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-344fc6da-lpnvp"]
Nov 03 19:39:29.932120 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.932087    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae213f75-435d-4d82-98b3-4dd6dbd78387" path="/var/lib/kubelet/pods/ae213f75-435d-4d82-98b3-4dd6dbd78387/volumes"
Nov 03 19:39:29.932476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:29.932463    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60ef5c9-714b-45c6-b418-6f31205635c5" path="/var/lib/kubelet/pods/d60ef5c9-714b-45c6-b418-6f31205635c5/volumes"
Nov 03 19:39:35.224085 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:39:35.224049    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:39:35.687781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:35.687745    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:39:39.806601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.806565    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"]
Nov 03 19:39:39.809668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.809651    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.858212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.858188    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:39:39.858212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.858198    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:39:39.894263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.894236    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sc6k\" (UniqueName: \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.894403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.894276    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.894403 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.894295    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.904348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.904316    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:39:39.904348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.904324    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:39:39.906051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.906032    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"]
Nov 03 19:39:39.995440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.995409    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.995594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.995452    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.995666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.995589    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sc6k\" (UniqueName: \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.996177 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.996155    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:39.997735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:39.997716    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:40.052957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.052927    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sc6k\" (UniqueName: \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\") pod \"maas-api-revoke-sub-c1e271a9-6g2mg\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:40.122952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.122881    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:40.492501 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.492470    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"]
Nov 03 19:39:40.499495 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:39:40.499465    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod681f82f2_5308_456f_bcfa_0a932326c368.slice/crio-7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee WatchSource:0}: Error finding container 7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee: Status 404 returned error can't find the container with id 7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee
Nov 03 19:39:40.832229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.832191    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg" event={"ID":"681f82f2-5308-456f-bcfa-0a932326c368","Type":"ContainerStarted","Data":"ce5f69d1566c60be81903b20428f05ef85a1d04e63b9f0fbbd3b2548c5572687"}
Nov 03 19:39:40.832577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.832261    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg" event={"ID":"681f82f2-5308-456f-bcfa-0a932326c368","Type":"ContainerStarted","Data":"7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee"}
Nov 03 19:39:40.940421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:40.940364    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg" podStartSLOduration=1.940350075 podStartE2EDuration="1.940350075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:39:40.939366054 +0000 UTC m=+3279.216719638" watchObservedRunningTime="2026-11-03 19:39:40.940350075 +0000 UTC m=+3279.217703658"
Nov 03 19:39:41.838772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:41.838738    4944 generic.go:356] "Generic (PLEG): container finished" podID="681f82f2-5308-456f-bcfa-0a932326c368" containerID="ce5f69d1566c60be81903b20428f05ef85a1d04e63b9f0fbbd3b2548c5572687" exitCode=0
Nov 03 19:39:41.839198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:41.838818    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg" event={"ID":"681f82f2-5308-456f-bcfa-0a932326c368","Type":"ContainerDied","Data":"ce5f69d1566c60be81903b20428f05ef85a1d04e63b9f0fbbd3b2548c5572687"}
Nov 03 19:39:42.975526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:42.975501    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:43.024020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.023984    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\") pod \"681f82f2-5308-456f-bcfa-0a932326c368\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") "
Nov 03 19:39:43.024161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.024047    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\") pod \"681f82f2-5308-456f-bcfa-0a932326c368\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") "
Nov 03 19:39:43.024161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.024096    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\" (UniqueName: \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\") pod \"681f82f2-5308-456f-bcfa-0a932326c368\" (UID: \"681f82f2-5308-456f-bcfa-0a932326c368\") "
Nov 03 19:39:43.024484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.024457    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca" pod "681f82f2-5308-456f-bcfa-0a932326c368" (UID: "681f82f2-5308-456f-bcfa-0a932326c368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:39:43.026287 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.026264    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k" pod "681f82f2-5308-456f-bcfa-0a932326c368" (UID: "681f82f2-5308-456f-bcfa-0a932326c368"). InnerVolumeSpecName "kube-api-access-2sc6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:39:43.026376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.026264    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca" pod "681f82f2-5308-456f-bcfa-0a932326c368" (UID: "681f82f2-5308-456f-bcfa-0a932326c368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:39:43.125916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.125844    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sc6k\" (UniqueName: \"kubernetes.io/projected/681f82f2-5308-456f-bcfa-0a932326c368-kube-api-access-2sc6k\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:43.125916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.125869    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:43.125916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.125879    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/681f82f2-5308-456f-bcfa-0a932326c368-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:39:43.849178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.849147    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"
Nov 03 19:39:43.849361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.849193    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg" event={"ID":"681f82f2-5308-456f-bcfa-0a932326c368","Type":"ContainerDied","Data":"7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee"}
Nov 03 19:39:43.849361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:43.849264    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cb79c1e978747f9f88d41b34a4b56f65143687918e81085548bfb9ee92ac5ee"
Nov 03 19:39:59.759844 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.759806    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"]
Nov 03 19:39:59.761568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.761538    4944 cpu_manager.go:414] "containerMap: removing container" podUID="681f82f2-5308-456f-bcfa-0a932326c368" containerName="revoke-keys"
Nov 03 19:39:59.761745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.761728    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="681f82f2-5308-456f-bcfa-0a932326c368" containerName="revoke-keys"
Nov 03 19:39:59.761993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.761975    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="681f82f2-5308-456f-bcfa-0a932326c368" containerName="revoke-keys"
Nov 03 19:39:59.765965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.765930    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.863688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.863602    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"]
Nov 03 19:39:59.877538 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877505    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877551    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877578    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877712    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877755    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkln\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-kube-api-access-8vkln\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877784    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.877948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877924    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.878074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.877981    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c38327-6415-43dc-b53b-8d2033f1bd4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.878113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.878078    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978746 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978715    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978770    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978826    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978846    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978867    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978900    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkln\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-kube-api-access-8vkln\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978926    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.978977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.978966    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979012    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c38327-6415-43dc-b53b-8d2033f1bd4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979206    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979258    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979483    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979545    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.979734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.979715    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c38327-6415-43dc-b53b-8d2033f1bd4e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.981400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.981380    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:39:59.981977 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:39:59.981958    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:00.037097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.037013    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:00.037281 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.037216    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkln\" (UniqueName: \"kubernetes.io/projected/55c38327-6415-43dc-b53b-8d2033f1bd4e-kube-api-access-8vkln\") pod \"maas-default-gateway-openshift-default-77f556d647-gwxrp\" (UID: \"55c38327-6415-43dc-b53b-8d2033f1bd4e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:00.082463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.082419    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:00.525637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.525597    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"]
Nov 03 19:40:00.531450 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:40:00.531421    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55c38327_6415_43dc_b53b_8d2033f1bd4e.slice/crio-d5abf1f079d3ee8e904811832a07fbfc3b404951ff712aa2a8c5970ec21d32ef WatchSource:0}: Error finding container d5abf1f079d3ee8e904811832a07fbfc3b404951ff712aa2a8c5970ec21d32ef: Status 404 returned error can't find the container with id d5abf1f079d3ee8e904811832a07fbfc3b404951ff712aa2a8c5970ec21d32ef
Nov 03 19:40:00.533544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.533516    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:00.533627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.533583    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:00.533627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.533613    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:40:00.941411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.941326    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" event={"ID":"55c38327-6415-43dc-b53b-8d2033f1bd4e","Type":"ContainerStarted","Data":"80ec318513ab22e088be4f6a26da857ddcf6db374517ea4124f55e59ff6d4f6b"}
Nov 03 19:40:00.941411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:00.941390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" event={"ID":"55c38327-6415-43dc-b53b-8d2033f1bd4e","Type":"ContainerStarted","Data":"d5abf1f079d3ee8e904811832a07fbfc3b404951ff712aa2a8c5970ec21d32ef"}
Nov 03 19:40:01.058331 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:01.058277    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podStartSLOduration=2.058261404 podStartE2EDuration="2.058261404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:01.056879298 +0000 UTC m=+3299.334232882" watchObservedRunningTime="2026-11-03 19:40:01.058261404 +0000 UTC m=+3299.335614991"
Nov 03 19:40:01.083220 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:01.083188    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:01.084751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:01.084722    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:01.084906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:01.084822    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:02.083714 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:02.083674    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:02.084125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:02.083740    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:02.451552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:02.451463    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:40:02.452510 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:02.452493    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:40:03.083719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:03.083683    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:03.084150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:03.083737    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:04.083355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:04.083321    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:04.083541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:04.083383    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:05.083499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.083457    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:05.084022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.083516    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:05.242169 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:40:05.242130    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:40:05.594354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.594321    4944 scope.go:122] "RemoveContainer" containerID="5314e44d2a36331b81658c018f089dbda4b85fa0bba038bcc83cebeb7a77f1b5"
Nov 03 19:40:05.602954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.602935    4944 scope.go:122] "RemoveContainer" containerID="1572c36546352c277c7ac07831f1647850c3b63dff336274d39558806ccba94b"
Nov 03 19:40:05.636694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.636667    4944 scope.go:122] "RemoveContainer" containerID="2cc7638dde390c94abaf1533a61fef9d13bd30e3c9511b31c8cc068f28efd1fa"
Nov 03 19:40:05.648450 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:05.648420    4944 scope.go:122] "RemoveContainer" containerID="27137cfc288c6074185a97e73835029112cdaf4ff7d5a67b8132af3266e9df51"
Nov 03 19:40:06.083147 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:06.083110    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:06.083330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:06.083164    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:07.083524 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:07.083478    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body=
Nov 03 19:40:07.083962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:07.083557    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused"
Nov 03 19:40:08.697909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:08.697870    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:08.698372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:08.698227    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:08.699022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:08.698996    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp"
Nov 03 19:40:11.382686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.382655    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"]
Nov 03 19:40:11.393564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.393531    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.440636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.440597    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"]
Nov 03 19:40:11.484570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.484539    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:40:11.484570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.484553    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:40:11.488185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.488168    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:40:11.488440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.488427    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:40:11.513890 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.513860    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.514030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.513995    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.514070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.514046    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g7xz\" (UniqueName: \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.615511 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.615470    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g7xz\" (UniqueName: \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.615717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.615556    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.615717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.615661    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.616254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.616226    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.618162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.618141    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.672583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.672512    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g7xz\" (UniqueName: \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\") pod \"maas-api-revoke-sub-2e1f5b5f-tdsnw\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:11.707114 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:11.707089    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:12.073377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:12.073341    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"]
Nov 03 19:40:12.079854 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:40:12.079828    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb089f3e1_8a27_4dbe_82cd_eb3e31885d9e.slice/crio-a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2 WatchSource:0}: Error finding container a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2: Status 404 returned error can't find the container with id a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2
Nov 03 19:40:13.004637 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:13.004598    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw" event={"ID":"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e","Type":"ContainerStarted","Data":"76956c4d098b569ff85cc7c5a5665895f53bdaf6db75fe73db448f1c170e9373"}
Nov 03 19:40:13.005047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:13.004658    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw" event={"ID":"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e","Type":"ContainerStarted","Data":"a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2"}
Nov 03 19:40:13.114515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:13.114460    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw" podStartSLOduration=2.114442926 podStartE2EDuration="2.114442926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:13.113146922 +0000 UTC m=+3311.390500505" watchObservedRunningTime="2026-11-03 19:40:13.114442926 +0000 UTC m=+3311.391796509"
Nov 03 19:40:14.010444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:14.010406    4944 generic.go:356] "Generic (PLEG): container finished" podID="b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" containerID="76956c4d098b569ff85cc7c5a5665895f53bdaf6db75fe73db448f1c170e9373" exitCode=0
Nov 03 19:40:14.010926 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:14.010441    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw" event={"ID":"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e","Type":"ContainerDied","Data":"76956c4d098b569ff85cc7c5a5665895f53bdaf6db75fe73db448f1c170e9373"}
Nov 03 19:40:15.153686 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.153656    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:15.347436 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.347356    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\") pod \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") "
Nov 03 19:40:15.347583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.347466    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\" (UniqueName: \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\") pod \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") "
Nov 03 19:40:15.347583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.347489    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\") pod \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\" (UID: \"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e\") "
Nov 03 19:40:15.347935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.347913    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca" pod "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" (UID: "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:15.349383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.349350    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca" pod "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" (UID: "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:15.349515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.349453    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz" pod "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" (UID: "b089f3e1-8a27-4dbe-82cd-eb3e31885d9e"). InnerVolumeSpecName "kube-api-access-4g7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:15.448886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.448854    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:15.448886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.448882    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g7xz\" (UniqueName: \"kubernetes.io/projected/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-kube-api-access-4g7xz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:15.449085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:15.448894    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:16.021541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:16.021512    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"
Nov 03 19:40:16.021541 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:16.021532    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw" event={"ID":"b089f3e1-8a27-4dbe-82cd-eb3e31885d9e","Type":"ContainerDied","Data":"a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2"}
Nov 03 19:40:16.021724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:16.021578    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a718dce96cde36ff6f3f9de470363bd0346214b808a900fe86e4b333cf5fa9d2"
Nov 03 19:40:23.932690 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.932660    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"]
Nov 03 19:40:23.933181 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.933164    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" containerName="revoke-keys"
Nov 03 19:40:23.933227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.933183    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" containerName="revoke-keys"
Nov 03 19:40:23.933271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.933261    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" containerName="revoke-keys"
Nov 03 19:40:23.936352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.936335    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:23.988836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:23.988781    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"]
Nov 03 19:40:24.018846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.018815    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.019018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.018858    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr45g\" (UniqueName: \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.019018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.018933    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.032769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.032741    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:40:24.032769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.032756    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:40:24.032769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.032756    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:40:24.033016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.032901    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:40:24.120476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.120439    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.120646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.120485    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr45g\" (UniqueName: \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.120646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.120575    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.121234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.121211    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.122976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.122959    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.178322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.178295    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr45g\" (UniqueName: \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\") pod \"maas-api-revoke-sub-ec843018-dcjfh\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.248772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.248742    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:24.622978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:24.622895    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"]
Nov 03 19:40:24.628773 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:40:24.628745    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df5f613_5139_48c7_8cfe_21de432ebb77.slice/crio-e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b WatchSource:0}: Error finding container e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b: Status 404 returned error can't find the container with id e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b
Nov 03 19:40:25.067254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:25.067216    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh" event={"ID":"4df5f613-5139-48c7-8cfe-21de432ebb77","Type":"ContainerStarted","Data":"0a7370a6c08e1db0c9a82c16ddd97bad93822ea61e24080ae91299de1f1bf96c"}
Nov 03 19:40:25.067643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:25.067276    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh" event={"ID":"4df5f613-5139-48c7-8cfe-21de432ebb77","Type":"ContainerStarted","Data":"e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b"}
Nov 03 19:40:25.176458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:25.176352    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh" podStartSLOduration=2.176337497 podStartE2EDuration="2.176337497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:40:25.174623064 +0000 UTC m=+3323.451976647" watchObservedRunningTime="2026-11-03 19:40:25.176337497 +0000 UTC m=+3323.453691080"
Nov 03 19:40:26.075225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:26.075189    4944 generic.go:356] "Generic (PLEG): container finished" podID="4df5f613-5139-48c7-8cfe-21de432ebb77" containerID="0a7370a6c08e1db0c9a82c16ddd97bad93822ea61e24080ae91299de1f1bf96c" exitCode=0
Nov 03 19:40:26.075683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:26.075279    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh" event={"ID":"4df5f613-5139-48c7-8cfe-21de432ebb77","Type":"ContainerDied","Data":"0a7370a6c08e1db0c9a82c16ddd97bad93822ea61e24080ae91299de1f1bf96c"}
Nov 03 19:40:27.203543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.203517    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:27.248922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.248890    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\" (UniqueName: \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\") pod \"4df5f613-5139-48c7-8cfe-21de432ebb77\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") "
Nov 03 19:40:27.249078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.248993    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\") pod \"4df5f613-5139-48c7-8cfe-21de432ebb77\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") "
Nov 03 19:40:27.249078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.249062    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\") pod \"4df5f613-5139-48c7-8cfe-21de432ebb77\" (UID: \"4df5f613-5139-48c7-8cfe-21de432ebb77\") "
Nov 03 19:40:27.249334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.249310    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca" pod "4df5f613-5139-48c7-8cfe-21de432ebb77" (UID: "4df5f613-5139-48c7-8cfe-21de432ebb77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:40:27.250970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.250940    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca" pod "4df5f613-5139-48c7-8cfe-21de432ebb77" (UID: "4df5f613-5139-48c7-8cfe-21de432ebb77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:40:27.251064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.251042    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g" pod "4df5f613-5139-48c7-8cfe-21de432ebb77" (UID: "4df5f613-5139-48c7-8cfe-21de432ebb77"). InnerVolumeSpecName "kube-api-access-gr45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:40:27.350675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.350594    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:27.350675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.350626    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr45g\" (UniqueName: \"kubernetes.io/projected/4df5f613-5139-48c7-8cfe-21de432ebb77-kube-api-access-gr45g\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:27.350675 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:27.350637    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4df5f613-5139-48c7-8cfe-21de432ebb77-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:40:28.086388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:28.086360    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"
Nov 03 19:40:28.086559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:28.086363    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh" event={"ID":"4df5f613-5139-48c7-8cfe-21de432ebb77","Type":"ContainerDied","Data":"e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b"}
Nov 03 19:40:28.086559 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:28.086479    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49dafaad36df751c95e6957dfd593be648dd21de678813d2359ed39dd6c349b"
Nov 03 19:40:35.223777 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:40:35.223743    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:40:36.015920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:36.015869    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:51.591032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:51.590994    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:51.883432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:51.883351    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:40:52.043264 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:52.043226    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"]
Nov 03 19:40:52.089064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:52.089026    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82f98dde-lvmqd"]
Nov 03 19:40:53.932382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:53.932348    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57de78a5-24e4-4789-ac31-fa31eed49dfa" path="/var/lib/kubelet/pods/57de78a5-24e4-4789-ac31-fa31eed49dfa/volumes"
Nov 03 19:40:59.198508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.198471    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"]
Nov 03 19:40:59.199031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.198853    4944 cpu_manager.go:414] "containerMap: removing container" podUID="4df5f613-5139-48c7-8cfe-21de432ebb77" containerName="revoke-keys"
Nov 03 19:40:59.199031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.198866    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4df5f613-5139-48c7-8cfe-21de432ebb77" containerName="revoke-keys"
Nov 03 19:40:59.199031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.198955    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="4df5f613-5139-48c7-8cfe-21de432ebb77" containerName="revoke-keys"
Nov 03 19:40:59.203153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.203134    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.288578 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.288546    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrw6\" (UniqueName: \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.288811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.288607    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.288811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.288688    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.300179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.300151    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:40:59.300179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.300182    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:40:59.300358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.300219    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:40:59.300358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.300261    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:40:59.302140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.302121    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"]
Nov 03 19:40:59.389522 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.389491    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrw6\" (UniqueName: \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.389658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.389633    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.389702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.389658    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.390406 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.390381    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.392080 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.392057    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.446554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.446527    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrw6\" (UniqueName: \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\") pod \"maas-api-revoke-sub-046e1343-bj2cj\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.515313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.515280    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:40:59.884207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:40:59.884130    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"]
Nov 03 19:40:59.890943 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:40:59.890916    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32cdd9c_19bb_4cc9_bba7_a585f5a0819d.slice/crio-c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96 WatchSource:0}: Error finding container c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96: Status 404 returned error can't find the container with id c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96
Nov 03 19:41:00.260055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:00.260023    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj" event={"ID":"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d","Type":"ContainerStarted","Data":"88a9b197bc12941384db96cfc1a6fb1351f53ca0ab92fcf251b466a84b4d1b02"}
Nov 03 19:41:00.260379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:00.260077    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj" event={"ID":"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d","Type":"ContainerStarted","Data":"c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96"}
Nov 03 19:41:00.368397 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:00.368348    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj" podStartSLOduration=1.368335747 podStartE2EDuration="1.368335747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:00.367284139 +0000 UTC m=+3358.644637722" watchObservedRunningTime="2026-11-03 19:41:00.368335747 +0000 UTC m=+3358.645689330"
Nov 03 19:41:01.266036 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:01.265997    4944 generic.go:356] "Generic (PLEG): container finished" podID="d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" containerID="88a9b197bc12941384db96cfc1a6fb1351f53ca0ab92fcf251b466a84b4d1b02" exitCode=0
Nov 03 19:41:01.266424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:01.266041    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj" event={"ID":"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d","Type":"ContainerDied","Data":"88a9b197bc12941384db96cfc1a6fb1351f53ca0ab92fcf251b466a84b4d1b02"}
Nov 03 19:41:02.414414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.414378    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:41:02.515376 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.515341    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\") pod \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") "
Nov 03 19:41:02.515551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.515414    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\") pod \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") "
Nov 03 19:41:02.515551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.515503    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\" (UniqueName: \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\") pod \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\" (UID: \"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d\") "
Nov 03 19:41:02.515773 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.515739    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca" pod "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" (UID: "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:41:02.517573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.517545    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca" pod "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" (UID: "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:41:02.517692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.517569    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6" pod "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" (UID: "d32cdd9c-19bb-4cc9-bba7-a585f5a0819d"). InnerVolumeSpecName "kube-api-access-rwrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:02.616995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.616960    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:02.616995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.616989    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrw6\" (UniqueName: \"kubernetes.io/projected/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-kube-api-access-rwrw6\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:02.616995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:02.617001    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:03.276971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:03.276934    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"
Nov 03 19:41:03.277172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:03.276986    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj" event={"ID":"d32cdd9c-19bb-4cc9-bba7-a585f5a0819d","Type":"ContainerDied","Data":"c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96"}
Nov 03 19:41:03.277172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:03.277044    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7bb57e567639d63a5dcb9e49038cf5bb55e5471211ae46695a50c34e6ba7b96"
Nov 03 19:41:04.043876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:04.043844    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"]
Nov 03 19:41:04.090280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:04.090240    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2afa005b-m5rmw"]
Nov 03 19:41:05.224165 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:41:05.224129    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:41:05.678297 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:05.678266    4944 scope.go:122] "RemoveContainer" containerID="379d70e1b4d623f89828282ed322201e57b5fd4b9971c983c2b6761dfa44a9fb"
Nov 03 19:41:05.686656 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:05.686624    4944 scope.go:122] "RemoveContainer" containerID="767e0165516085163e0ff598a08148a157e186580e85f9812f8a94c5a1aa7fb5"
Nov 03 19:41:05.694392 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:05.694359    4944 scope.go:122] "RemoveContainer" containerID="8650918dae5c53fced0ca58ba4d7a13b34222448bceb87c88d9e9163b302538d"
Nov 03 19:41:05.701558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:05.701540    4944 scope.go:122] "RemoveContainer" containerID="f23c63ac2ce3fd22551bf185ad6a627a61a319edb25c4b8f29a9f84a39a382ef"
Nov 03 19:41:05.932935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:05.932862    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47159d18-da63-4c6a-b1fc-26b5a26cda6c" path="/var/lib/kubelet/pods/47159d18-da63-4c6a-b1fc-26b5a26cda6c/volumes"
Nov 03 19:41:09.888451 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.888416    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"]
Nov 03 19:41:09.888855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.888800    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" containerName="revoke-keys"
Nov 03 19:41:09.888855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.888812    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" containerName="revoke-keys"
Nov 03 19:41:09.888946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.888892    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" containerName="revoke-keys"
Nov 03 19:41:09.893765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.893745    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:09.987842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.987808    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jwqh\" (UniqueName: \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:09.988018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.987930    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:09.988018 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.987968    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:09.989533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.989501    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:41:09.989658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.989536    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:41:09.989658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.989543    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:41:09.989658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.989630    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:41:09.991568 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:09.991549    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"]
Nov 03 19:41:10.089295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.089255    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jwqh\" (UniqueName: \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.089498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.089327    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.089498 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.089351    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.090031 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.090007    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.091678 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.091657    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.145999 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.145934    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jwqh\" (UniqueName: \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\") pod \"maas-api-revoke-sub-d3fba056-z6vsd\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.206632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.206589    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:10.576188 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:10.576150    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"]
Nov 03 19:41:10.582723 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:41:10.582684    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0dbea4_d04b_4559_83e9_4641dd2171d6.slice/crio-d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88 WatchSource:0}: Error finding container d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88: Status 404 returned error can't find the container with id d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88
Nov 03 19:41:11.317109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:11.317075    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd" event={"ID":"0d0dbea4-d04b-4559-83e9-4641dd2171d6","Type":"ContainerStarted","Data":"87ecc6e4af31bb55ec130c1efea658b284af01e359f87e126ef39fb61bd6bb2b"}
Nov 03 19:41:11.317484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:11.317138    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd" event={"ID":"0d0dbea4-d04b-4559-83e9-4641dd2171d6","Type":"ContainerStarted","Data":"d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88"}
Nov 03 19:41:11.425156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:11.425110    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd" podStartSLOduration=2.425094537 podStartE2EDuration="2.425094537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:11.424766768 +0000 UTC m=+3369.702120351" watchObservedRunningTime="2026-11-03 19:41:11.425094537 +0000 UTC m=+3369.702448121"
Nov 03 19:41:12.323570 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:12.323536    4944 generic.go:356] "Generic (PLEG): container finished" podID="0d0dbea4-d04b-4559-83e9-4641dd2171d6" containerID="87ecc6e4af31bb55ec130c1efea658b284af01e359f87e126ef39fb61bd6bb2b" exitCode=0
Nov 03 19:41:12.323971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:12.323596    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd" event={"ID":"0d0dbea4-d04b-4559-83e9-4641dd2171d6","Type":"ContainerDied","Data":"87ecc6e4af31bb55ec130c1efea658b284af01e359f87e126ef39fb61bd6bb2b"}
Nov 03 19:41:13.696165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.696129    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:13.876997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.873108    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\") pod \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") "
Nov 03 19:41:13.876997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.873204    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\" (UniqueName: \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\") pod \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") "
Nov 03 19:41:13.876997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.873283    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\") pod \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\" (UID: \"0d0dbea4-d04b-4559-83e9-4641dd2171d6\") "
Nov 03 19:41:13.876997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.874995    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca" pod "0d0dbea4-d04b-4559-83e9-4641dd2171d6" (UID: "0d0dbea4-d04b-4559-83e9-4641dd2171d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:41:13.881540 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.881246    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca" pod "0d0dbea4-d04b-4559-83e9-4641dd2171d6" (UID: "0d0dbea4-d04b-4559-83e9-4641dd2171d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:41:13.883856 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.883820    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh" pod "0d0dbea4-d04b-4559-83e9-4641dd2171d6" (UID: "0d0dbea4-d04b-4559-83e9-4641dd2171d6"). InnerVolumeSpecName "kube-api-access-8jwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:13.974249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.974211    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:13.974518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.974498    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jwqh\" (UniqueName: \"kubernetes.io/projected/0d0dbea4-d04b-4559-83e9-4641dd2171d6-kube-api-access-8jwqh\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:13.974638 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:13.974624    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d0dbea4-d04b-4559-83e9-4641dd2171d6-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:14.369813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:14.369316    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd" event={"ID":"0d0dbea4-d04b-4559-83e9-4641dd2171d6","Type":"ContainerDied","Data":"d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88"}
Nov 03 19:41:14.369813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:14.369380    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d032975fe67a5dd60624029198f27d1d8b0f0de52a3c905d39f28eb64c131c88"
Nov 03 19:41:14.369813 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:14.369452    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"
Nov 03 19:41:35.223924 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:41:35.223888    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:41:36.453431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:36.453393    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:41:45.242573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.242540    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"]
Nov 03 19:41:45.242987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.242973    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0d0dbea4-d04b-4559-83e9-4641dd2171d6" containerName="revoke-keys"
Nov 03 19:41:45.243034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.242987    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0dbea4-d04b-4559-83e9-4641dd2171d6" containerName="revoke-keys"
Nov 03 19:41:45.243072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.243055    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0dbea4-d04b-4559-83e9-4641dd2171d6" containerName="revoke-keys"
Nov 03 19:41:45.246223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.246196    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.295207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.295165    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:41:45.295207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.295190    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:41:45.341529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.341499    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:41:45.341694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.341572    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:41:45.343429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.343409    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"]
Nov 03 19:41:45.410861 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.410827    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.411017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.410957    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs8km\" (UniqueName: \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.411076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.411019    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.512068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.511980    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.512068 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.512060    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs8km\" (UniqueName: \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.512286 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.512100    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.512744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.512723    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.514415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.514388    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.568871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.568837    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs8km\" (UniqueName: \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\") pod \"maas-api-revoke-sub-ae594979-7wl7n\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:45.858749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:45.858664    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:46.224415 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:46.224324    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"]
Nov 03 19:41:46.231171 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:41:46.231145    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b79391_20f7_4e55_883b_5606e457917a.slice/crio-0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3 WatchSource:0}: Error finding container 0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3: Status 404 returned error can't find the container with id 0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3
Nov 03 19:41:46.541162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:46.541127    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n" event={"ID":"a7b79391-20f7-4e55-883b-5606e457917a","Type":"ContainerStarted","Data":"662db66cd39cbf57085f5cdb1e47af81303056f52b059ab03666e68179cefa43"}
Nov 03 19:41:46.541506 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:46.541190    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n" event={"ID":"a7b79391-20f7-4e55-883b-5606e457917a","Type":"ContainerStarted","Data":"0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3"}
Nov 03 19:41:46.649295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:46.649249    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n" podStartSLOduration=1.6492353149999999 podStartE2EDuration="1.649235315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:41:46.649115234 +0000 UTC m=+3404.926468827" watchObservedRunningTime="2026-11-03 19:41:46.649235315 +0000 UTC m=+3404.926588900"
Nov 03 19:41:47.550161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:47.550121    4944 generic.go:356] "Generic (PLEG): container finished" podID="a7b79391-20f7-4e55-883b-5606e457917a" containerID="662db66cd39cbf57085f5cdb1e47af81303056f52b059ab03666e68179cefa43" exitCode=0
Nov 03 19:41:47.550598 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:47.550203    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n" event={"ID":"a7b79391-20f7-4e55-883b-5606e457917a","Type":"ContainerDied","Data":"662db66cd39cbf57085f5cdb1e47af81303056f52b059ab03666e68179cefa43"}
Nov 03 19:41:48.686037 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.686008    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:48.846316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.846242    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\" (UniqueName: \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\") pod \"a7b79391-20f7-4e55-883b-5606e457917a\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") "
Nov 03 19:41:48.846465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.846314    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\") pod \"a7b79391-20f7-4e55-883b-5606e457917a\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") "
Nov 03 19:41:48.846465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.846422    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\") pod \"a7b79391-20f7-4e55-883b-5606e457917a\" (UID: \"a7b79391-20f7-4e55-883b-5606e457917a\") "
Nov 03 19:41:48.846659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.846638    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca" pod "a7b79391-20f7-4e55-883b-5606e457917a" (UID: "a7b79391-20f7-4e55-883b-5606e457917a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:41:48.846755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.846734    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:48.848360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.848338    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km" pod "a7b79391-20f7-4e55-883b-5606e457917a" (UID: "a7b79391-20f7-4e55-883b-5606e457917a"). InnerVolumeSpecName "kube-api-access-qs8km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:41:48.848483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.848396    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca" pod "a7b79391-20f7-4e55-883b-5606e457917a" (UID: "a7b79391-20f7-4e55-883b-5606e457917a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:41:48.947257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.947228    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7b79391-20f7-4e55-883b-5606e457917a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:48.947257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:48.947253    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs8km\" (UniqueName: \"kubernetes.io/projected/a7b79391-20f7-4e55-883b-5606e457917a-kube-api-access-qs8km\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:41:49.562051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:49.562017    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n" event={"ID":"a7b79391-20f7-4e55-883b-5606e457917a","Type":"ContainerDied","Data":"0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3"}
Nov 03 19:41:49.562051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:49.562046    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"
Nov 03 19:41:49.562280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:41:49.562063    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d09604daa3e2ea7262ffdc6908855ce090137ef47dd37bd1884279dbcbf0bc3"
Nov 03 19:42:05.283660 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:42:05.283568    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:42:05.743167 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:05.743140    4944 scope.go:122] "RemoveContainer" containerID="13df82360fadd2ae05c86dea4dd1498cde2e4231146b18494f870053e569e6f1"
Nov 03 19:42:05.775753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:05.775612    4944 scope.go:122] "RemoveContainer" containerID="c97f7d57789fa1e6dfe8e9f3991faf1510899859d1e677ae0e9e199fd0b8e75c"
Nov 03 19:42:13.628125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.628091    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"]
Nov 03 19:42:13.628657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.628509    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a7b79391-20f7-4e55-883b-5606e457917a" containerName="revoke-keys"
Nov 03 19:42:13.628657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.628524    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7b79391-20f7-4e55-883b-5606e457917a" containerName="revoke-keys"
Nov 03 19:42:13.628657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.628607    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7b79391-20f7-4e55-883b-5606e457917a" containerName="revoke-keys"
Nov 03 19:42:13.633449 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.633424    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.682255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.682220    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.682255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.682255    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:42:13.682507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.682354    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.682507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.682407    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkmwx\" (UniqueName: \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.728721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.728675    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:42:13.728721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.728702    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:42:13.728721 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.728681    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:42:13.730667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.730649    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"]
Nov 03 19:42:13.783312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.783284    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.783472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.783342    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.783472 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.783380    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkmwx\" (UniqueName: \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.784024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.784006    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.785448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.785417    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.841198 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.841173    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkmwx\" (UniqueName: \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\") pod \"maas-api-revoke-sub-a4aef7d7-m9whs\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:13.947142 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:13.947066    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:14.315520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:14.315477    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"]
Nov 03 19:42:14.321843 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:42:14.321812    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5996061c_2522_4b68_9ba5_0a3bde033e9d.slice/crio-4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0 WatchSource:0}: Error finding container 4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0: Status 404 returned error can't find the container with id 4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0
Nov 03 19:42:14.725633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:14.725599    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs" event={"ID":"5996061c-2522-4b68-9ba5-0a3bde033e9d","Type":"ContainerStarted","Data":"8673a626980b9293b4bbccbe982fa1467de225f5421233da81127f62918e0d85"}
Nov 03 19:42:14.726089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:14.725657    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs" event={"ID":"5996061c-2522-4b68-9ba5-0a3bde033e9d","Type":"ContainerStarted","Data":"4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0"}
Nov 03 19:42:14.836025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:14.835962    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs" podStartSLOduration=1.835946399 podStartE2EDuration="1.835946399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:14.83431429 +0000 UTC m=+3433.111667876" watchObservedRunningTime="2026-11-03 19:42:14.835946399 +0000 UTC m=+3433.113299989"
Nov 03 19:42:15.732030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:15.731998    4944 generic.go:356] "Generic (PLEG): container finished" podID="5996061c-2522-4b68-9ba5-0a3bde033e9d" containerID="8673a626980b9293b4bbccbe982fa1467de225f5421233da81127f62918e0d85" exitCode=0
Nov 03 19:42:15.732030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:15.732035    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs" event={"ID":"5996061c-2522-4b68-9ba5-0a3bde033e9d","Type":"ContainerDied","Data":"8673a626980b9293b4bbccbe982fa1467de225f5421233da81127f62918e0d85"}
Nov 03 19:42:16.863345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.863319    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:16.913213 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.913185    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\") pod \"5996061c-2522-4b68-9ba5-0a3bde033e9d\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") "
Nov 03 19:42:16.913365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.913220    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\" (UniqueName: \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\") pod \"5996061c-2522-4b68-9ba5-0a3bde033e9d\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") "
Nov 03 19:42:16.913365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.913267    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\") pod \"5996061c-2522-4b68-9ba5-0a3bde033e9d\" (UID: \"5996061c-2522-4b68-9ba5-0a3bde033e9d\") "
Nov 03 19:42:16.913573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.913546    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca" pod "5996061c-2522-4b68-9ba5-0a3bde033e9d" (UID: "5996061c-2522-4b68-9ba5-0a3bde033e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:42:16.915303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.915283    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx" pod "5996061c-2522-4b68-9ba5-0a3bde033e9d" (UID: "5996061c-2522-4b68-9ba5-0a3bde033e9d"). InnerVolumeSpecName "kube-api-access-tkmwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:42:16.915377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:16.915297    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca" pod "5996061c-2522-4b68-9ba5-0a3bde033e9d" (UID: "5996061c-2522-4b68-9ba5-0a3bde033e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:42:17.014544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.014509    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:17.014544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.014536    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkmwx\" (UniqueName: \"kubernetes.io/projected/5996061c-2522-4b68-9ba5-0a3bde033e9d-kube-api-access-tkmwx\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:17.014544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.014548    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5996061c-2522-4b68-9ba5-0a3bde033e9d-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:17.742244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.742213    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"
Nov 03 19:42:17.742422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.742249    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs" event={"ID":"5996061c-2522-4b68-9ba5-0a3bde033e9d","Type":"ContainerDied","Data":"4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0"}
Nov 03 19:42:17.742422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:17.742294    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b5deae0e0e15103ce42bbe5535b0f0df18321d7ebc57fb64bd37cebe2b9aec0"
Nov 03 19:42:35.284635 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:42:35.284589    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:42:36.754723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:36.754684    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:38.892277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:38.892241    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:42:43.932988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.932915    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"]
Nov 03 19:42:43.933349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.933317    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5996061c-2522-4b68-9ba5-0a3bde033e9d" containerName="revoke-keys"
Nov 03 19:42:43.933349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.933331    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5996061c-2522-4b68-9ba5-0a3bde033e9d" containerName="revoke-keys"
Nov 03 19:42:43.933434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.933408    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5996061c-2522-4b68-9ba5-0a3bde033e9d" containerName="revoke-keys"
Nov 03 19:42:43.936432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.936414    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:43.965635 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.965602    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:43.965819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.965653    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfllq\" (UniqueName: \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:43.965819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.965675    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:43.985504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:43.985470    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:42:44.032456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.032421    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:42:44.032616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.032480    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:42:44.034483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.034460    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"]
Nov 03 19:42:44.040752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.040734    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:42:44.066708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.066682    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.066850 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.066730    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfllq\" (UniqueName: \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.066898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.066849    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.067456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.067433    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.069140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.069122    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.124216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.124183    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfllq\" (UniqueName: \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\") pod \"maas-api-revoke-sub-79805ca9-s9b6s\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.253831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.253772    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:44.622808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.622688    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"]
Nov 03 19:42:44.628438 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:42:44.628410    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8774b553_0ec0_43a3_961a_80ddc3d4e94f.slice/crio-d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b WatchSource:0}: Error finding container d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b: Status 404 returned error can't find the container with id d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b
Nov 03 19:42:44.886443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.886379    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s" event={"ID":"8774b553-0ec0-43a3-961a-80ddc3d4e94f","Type":"ContainerStarted","Data":"5782cd667f105a44c20cf307fca10f2d498e94c57de70e561807f4b7c01cb668"}
Nov 03 19:42:44.886443 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.886428    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s" event={"ID":"8774b553-0ec0-43a3-961a-80ddc3d4e94f","Type":"ContainerStarted","Data":"d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b"}
Nov 03 19:42:44.996441 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:44.996389    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s" podStartSLOduration=1.996367894 podStartE2EDuration="1.996367894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:42:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:42:44.995099923 +0000 UTC m=+3463.272453506" watchObservedRunningTime="2026-11-03 19:42:44.996367894 +0000 UTC m=+3463.273721477"
Nov 03 19:42:45.892504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:45.892467    4944 generic.go:356] "Generic (PLEG): container finished" podID="8774b553-0ec0-43a3-961a-80ddc3d4e94f" containerID="5782cd667f105a44c20cf307fca10f2d498e94c57de70e561807f4b7c01cb668" exitCode=0
Nov 03 19:42:45.892711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:45.892529    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s" event={"ID":"8774b553-0ec0-43a3-961a-80ddc3d4e94f","Type":"ContainerDied","Data":"5782cd667f105a44c20cf307fca10f2d498e94c57de70e561807f4b7c01cb668"}
Nov 03 19:42:47.030312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.030288    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:47.096429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.096395    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\" (UniqueName: \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\") pod \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") "
Nov 03 19:42:47.096604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.096590    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\") pod \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") "
Nov 03 19:42:47.096650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.096616    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\") pod \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\" (UID: \"8774b553-0ec0-43a3-961a-80ddc3d4e94f\") "
Nov 03 19:42:47.097070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.097042    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca" pod "8774b553-0ec0-43a3-961a-80ddc3d4e94f" (UID: "8774b553-0ec0-43a3-961a-80ddc3d4e94f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:42:47.098579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.098548    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq" pod "8774b553-0ec0-43a3-961a-80ddc3d4e94f" (UID: "8774b553-0ec0-43a3-961a-80ddc3d4e94f"). InnerVolumeSpecName "kube-api-access-vfllq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:42:47.098666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.098595    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca" pod "8774b553-0ec0-43a3-961a-80ddc3d4e94f" (UID: "8774b553-0ec0-43a3-961a-80ddc3d4e94f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:42:47.197447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.197368    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfllq\" (UniqueName: \"kubernetes.io/projected/8774b553-0ec0-43a3-961a-80ddc3d4e94f-kube-api-access-vfllq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:47.197447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.197412    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:47.197447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.197423    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8774b553-0ec0-43a3-961a-80ddc3d4e94f-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:42:47.905456 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.905425    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"
Nov 03 19:42:47.905688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.905452    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s" event={"ID":"8774b553-0ec0-43a3-961a-80ddc3d4e94f","Type":"ContainerDied","Data":"d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b"}
Nov 03 19:42:47.905688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:47.905504    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d44efbf832429036de7175e791dda821e8a3af7d7be3ab26187c005bae176a4b"
Nov 03 19:42:53.050489 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:53.050442    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"]
Nov 03 19:42:53.096722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:53.096682    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61cef7d1-75rkn"]
Nov 03 19:42:53.932868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:42:53.932828    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cc9e66-3441-46b4-9c8b-1fbed3215a8b" path="/var/lib/kubelet/pods/f8cc9e66-3441-46b4-9c8b-1fbed3215a8b/volumes"
Nov 03 19:43:04.550355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.550318    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"]
Nov 03 19:43:04.550749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.550734    4944 cpu_manager.go:414] "containerMap: removing container" podUID="8774b553-0ec0-43a3-961a-80ddc3d4e94f" containerName="revoke-keys"
Nov 03 19:43:04.550837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.550750    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8774b553-0ec0-43a3-961a-80ddc3d4e94f" containerName="revoke-keys"
Nov 03 19:43:04.550874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.550855    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="8774b553-0ec0-43a3-961a-80ddc3d4e94f" containerName="revoke-keys"
Nov 03 19:43:04.553760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.553739    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.602520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.602498    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:04.602702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.602682    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:43:04.602759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.602747    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:43:04.648891 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.648860    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:43:04.650995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.650969    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"]
Nov 03 19:43:04.693500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.693471    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq289\" (UniqueName: \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.693622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.693506    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.693674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.693627    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.794692 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.794661    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq289\" (UniqueName: \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.794863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.794699    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.794863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.794762    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.795422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.795399    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.797074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.797056    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.851881 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.851815    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq289\" (UniqueName: \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\") pod \"maas-api-revoke-sub-0b7e8edb-4k2fw\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:04.865507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:04.865487    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:05.223750 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:43:05.223663    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:43:05.253378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:05.253353    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"]
Nov 03 19:43:05.259345 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:43:05.259318    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c1ba47_4490_46cd_b5b2_8081207554c9.slice/crio-f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e WatchSource:0}: Error finding container f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e: Status 404 returned error can't find the container with id f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e
Nov 03 19:43:05.807263 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:05.807232    4944 scope.go:122] "RemoveContainer" containerID="0638434e7237524330c9bdd24100d5a34b9491529545e47b051ce8fc79c8f979"
Nov 03 19:43:05.815634 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:05.815617    4944 scope.go:122] "RemoveContainer" containerID="a92f593e14856b7b78781c860828d6e0b92b31dd103e2f71d89658bb96d54d40"
Nov 03 19:43:06.000474 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:06.000434    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw" event={"ID":"85c1ba47-4490-46cd-b5b2-8081207554c9","Type":"ContainerStarted","Data":"7df35c2200b4ca4e4772629072e3b5a89c7f2b276767c01e7c594c95648ea362"}
Nov 03 19:43:06.000644 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:06.000496    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw" event={"ID":"85c1ba47-4490-46cd-b5b2-8081207554c9","Type":"ContainerStarted","Data":"f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e"}
Nov 03 19:43:06.109924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:06.109829    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw" podStartSLOduration=2.109816374 podStartE2EDuration="2.109816374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:06.109162588 +0000 UTC m=+3484.386516172" watchObservedRunningTime="2026-11-03 19:43:06.109816374 +0000 UTC m=+3484.387169967"
Nov 03 19:43:07.007256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:07.007225    4944 generic.go:356] "Generic (PLEG): container finished" podID="85c1ba47-4490-46cd-b5b2-8081207554c9" containerID="7df35c2200b4ca4e4772629072e3b5a89c7f2b276767c01e7c594c95648ea362" exitCode=0
Nov 03 19:43:07.007654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:07.007287    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw" event={"ID":"85c1ba47-4490-46cd-b5b2-8081207554c9","Type":"ContainerDied","Data":"7df35c2200b4ca4e4772629072e3b5a89c7f2b276767c01e7c594c95648ea362"}
Nov 03 19:43:08.145975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.145947    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:08.327420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.327332    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\") pod \"85c1ba47-4490-46cd-b5b2-8081207554c9\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") "
Nov 03 19:43:08.327420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.327375    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\" (UniqueName: \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\") pod \"85c1ba47-4490-46cd-b5b2-8081207554c9\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") "
Nov 03 19:43:08.327420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.327408    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\") pod \"85c1ba47-4490-46cd-b5b2-8081207554c9\" (UID: \"85c1ba47-4490-46cd-b5b2-8081207554c9\") "
Nov 03 19:43:08.327766 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.327742    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca" pod "85c1ba47-4490-46cd-b5b2-8081207554c9" (UID: "85c1ba47-4490-46cd-b5b2-8081207554c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:08.329556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.329532    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289" pod "85c1ba47-4490-46cd-b5b2-8081207554c9" (UID: "85c1ba47-4490-46cd-b5b2-8081207554c9"). InnerVolumeSpecName "kube-api-access-kq289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:08.329655 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.329579    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca" pod "85c1ba47-4490-46cd-b5b2-8081207554c9" (UID: "85c1ba47-4490-46cd-b5b2-8081207554c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:08.428137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.428105    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:08.428137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.428131    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq289\" (UniqueName: \"kubernetes.io/projected/85c1ba47-4490-46cd-b5b2-8081207554c9-kube-api-access-kq289\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:08.428137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:08.428142    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85c1ba47-4490-46cd-b5b2-8081207554c9-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:09.017924 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:09.017892    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"
Nov 03 19:43:09.018113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:09.017921    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw" event={"ID":"85c1ba47-4490-46cd-b5b2-8081207554c9","Type":"ContainerDied","Data":"f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e"}
Nov 03 19:43:09.018113 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:09.017971    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2e834728850aee72e2bd46619a5423e7f021e0df19ca1d80b0a238475f2c40e"
Nov 03 19:43:15.786586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.786547    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"]
Nov 03 19:43:15.787226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.786982    4944 cpu_manager.go:414] "containerMap: removing container" podUID="85c1ba47-4490-46cd-b5b2-8081207554c9" containerName="revoke-keys"
Nov 03 19:43:15.787226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.786996    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85c1ba47-4490-46cd-b5b2-8081207554c9" containerName="revoke-keys"
Nov 03 19:43:15.787226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.787058    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="85c1ba47-4490-46cd-b5b2-8081207554c9" containerName="revoke-keys"
Nov 03 19:43:15.790226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.790206    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.790975 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.790947    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj2ml\" (UniqueName: \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.791107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.791005    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.791107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.791092    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.891672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.891633    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.891672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.891676    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.891954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.891730    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj2ml\" (UniqueName: \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.892112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.892094    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:43:15.892164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.892093    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:15.892164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.892130    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:43:15.892164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.892130    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:43:15.894642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.894622    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"]
Nov 03 19:43:15.903210 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.903192    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.904849 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.904826    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:15.949500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:15.949475    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj2ml\" (UniqueName: \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\") pod \"maas-api-revoke-sub-3d4156f1-d4dcf\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:16.103248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:16.103168    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:16.471751 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:16.471673    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"]
Nov 03 19:43:16.478269 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:43:16.478240    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944c7743_8839_4555_99cc_98f88d4329c6.slice/crio-c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e WatchSource:0}: Error finding container c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e: Status 404 returned error can't find the container with id c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e
Nov 03 19:43:17.062955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:17.062913    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf" event={"ID":"944c7743-8839-4555-99cc-98f88d4329c6","Type":"ContainerStarted","Data":"1d4d62ca8a13a18f29a71c7bc7a81a26db86320fc49c2f3ac8a920a16f5b9473"}
Nov 03 19:43:17.063362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:17.063002    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf" event={"ID":"944c7743-8839-4555-99cc-98f88d4329c6","Type":"ContainerStarted","Data":"c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e"}
Nov 03 19:43:17.172155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:17.172108    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf" podStartSLOduration=2.1720953339999998 podStartE2EDuration="2.172095334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:17.171415857 +0000 UTC m=+3495.448769439" watchObservedRunningTime="2026-11-03 19:43:17.172095334 +0000 UTC m=+3495.449448927"
Nov 03 19:43:18.069421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:18.069387    4944 generic.go:356] "Generic (PLEG): container finished" podID="944c7743-8839-4555-99cc-98f88d4329c6" containerID="1d4d62ca8a13a18f29a71c7bc7a81a26db86320fc49c2f3ac8a920a16f5b9473" exitCode=0
Nov 03 19:43:18.069825 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:18.069462    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf" event={"ID":"944c7743-8839-4555-99cc-98f88d4329c6","Type":"ContainerDied","Data":"1d4d62ca8a13a18f29a71c7bc7a81a26db86320fc49c2f3ac8a920a16f5b9473"}
Nov 03 19:43:19.201180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.201151    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:19.221358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.221331    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\" (UniqueName: \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\") pod \"944c7743-8839-4555-99cc-98f88d4329c6\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") "
Nov 03 19:43:19.221475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.221381    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\") pod \"944c7743-8839-4555-99cc-98f88d4329c6\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") "
Nov 03 19:43:19.221475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.221416    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\") pod \"944c7743-8839-4555-99cc-98f88d4329c6\" (UID: \"944c7743-8839-4555-99cc-98f88d4329c6\") "
Nov 03 19:43:19.221737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.221715    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca" pod "944c7743-8839-4555-99cc-98f88d4329c6" (UID: "944c7743-8839-4555-99cc-98f88d4329c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:19.223424 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.223401    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml" pod "944c7743-8839-4555-99cc-98f88d4329c6" (UID: "944c7743-8839-4555-99cc-98f88d4329c6"). InnerVolumeSpecName "kube-api-access-gj2ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:19.223492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.223400    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca" pod "944c7743-8839-4555-99cc-98f88d4329c6" (UID: "944c7743-8839-4555-99cc-98f88d4329c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:19.322726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.322694    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj2ml\" (UniqueName: \"kubernetes.io/projected/944c7743-8839-4555-99cc-98f88d4329c6-kube-api-access-gj2ml\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:19.322726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.322728    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:19.322931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:19.322742    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/944c7743-8839-4555-99cc-98f88d4329c6-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:20.080161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:20.080131    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"
Nov 03 19:43:20.080374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:20.080167    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf" event={"ID":"944c7743-8839-4555-99cc-98f88d4329c6","Type":"ContainerDied","Data":"c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e"}
Nov 03 19:43:20.080374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:20.080218    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c9fc6924f1282c4e88cbe8a1e17f1f7d0f0a1a7c85f65b4791247927ff589e"
Nov 03 19:43:32.706756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:32.706718    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:33.777715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.777684    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"]
Nov 03 19:43:33.824078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.824050    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d4dcf"]
Nov 03 19:43:33.904455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.904423    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"]
Nov 03 19:43:33.904834 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.904820    4944 cpu_manager.go:414] "containerMap: removing container" podUID="944c7743-8839-4555-99cc-98f88d4329c6" containerName="revoke-keys"
Nov 03 19:43:33.904896 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.904836    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944c7743-8839-4555-99cc-98f88d4329c6" containerName="revoke-keys"
Nov 03 19:43:33.904939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.904923    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="944c7743-8839-4555-99cc-98f88d4329c6" containerName="revoke-keys"
Nov 03 19:43:33.907870 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.907846    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:33.932661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.932626    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944c7743-8839-4555-99cc-98f88d4329c6" path="/var/lib/kubelet/pods/944c7743-8839-4555-99cc-98f88d4329c6/volumes"
Nov 03 19:43:33.956930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.956901    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:43:33.956930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.956923    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:43:33.957081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.956901    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:43:33.957081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.956924    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:33.959420 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.959396    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:33.959519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.959442    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:33.959612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.959595    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klth4\" (UniqueName: \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:33.967138 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:33.967114    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"]
Nov 03 19:43:34.060463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.060380    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klth4\" (UniqueName: \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.060463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.060437    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.060668 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.060471    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.061111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.061091    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.062835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.062802    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.128271 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.128238    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klth4\" (UniqueName: \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\") pod \"maas-api-revoke-sub-3d4156f1-b652g\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.220642 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.220610    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:34.590819 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:34.590760    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"]
Nov 03 19:43:34.597392 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:43:34.597360    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdde7c4f_52f9_4d74_aa06_be9f830bee8b.slice/crio-a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028 WatchSource:0}: Error finding container a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028: Status 404 returned error can't find the container with id a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028
Nov 03 19:43:35.153529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:35.153490    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g" event={"ID":"bdde7c4f-52f9-4d74-aa06-be9f830bee8b","Type":"ContainerStarted","Data":"c395b0ca5482b3487e87d4432c7aa68333c9809f7c70e035867b0814e0bf164e"}
Nov 03 19:43:35.153982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:35.153544    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g" event={"ID":"bdde7c4f-52f9-4d74-aa06-be9f830bee8b","Type":"ContainerStarted","Data":"a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028"}
Nov 03 19:43:35.224952 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:43:35.224870    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:43:35.271129 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:35.271077    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g" podStartSLOduration=2.271063177 podStartE2EDuration="2.271063177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:35.270465959 +0000 UTC m=+3513.547819554" watchObservedRunningTime="2026-11-03 19:43:35.271063177 +0000 UTC m=+3513.548416804"
Nov 03 19:43:36.159929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:36.159891    4944 generic.go:356] "Generic (PLEG): container finished" podID="bdde7c4f-52f9-4d74-aa06-be9f830bee8b" containerID="c395b0ca5482b3487e87d4432c7aa68333c9809f7c70e035867b0814e0bf164e" exitCode=0
Nov 03 19:43:36.159929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:36.159922    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g" event={"ID":"bdde7c4f-52f9-4d74-aa06-be9f830bee8b","Type":"ContainerDied","Data":"c395b0ca5482b3487e87d4432c7aa68333c9809f7c70e035867b0814e0bf164e"}
Nov 03 19:43:37.109418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.109377    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:37.300017 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.299988    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:37.376812 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.376676    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:43:37.395274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.395245    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\") pod \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") "
Nov 03 19:43:37.395431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.395311    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\") pod \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") "
Nov 03 19:43:37.395431 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.395346    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\" (UniqueName: \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\") pod \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\" (UID: \"bdde7c4f-52f9-4d74-aa06-be9f830bee8b\") "
Nov 03 19:43:37.395687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.395667    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca" pod "bdde7c4f-52f9-4d74-aa06-be9f830bee8b" (UID: "bdde7c4f-52f9-4d74-aa06-be9f830bee8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:37.397483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.397451    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4" pod "bdde7c4f-52f9-4d74-aa06-be9f830bee8b" (UID: "bdde7c4f-52f9-4d74-aa06-be9f830bee8b"). InnerVolumeSpecName "kube-api-access-klth4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:37.397609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.397531    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca" pod "bdde7c4f-52f9-4d74-aa06-be9f830bee8b" (UID: "bdde7c4f-52f9-4d74-aa06-be9f830bee8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:37.496499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.496464    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.496499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.496496    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:37.496728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:37.496512    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klth4\" (UniqueName: \"kubernetes.io/projected/bdde7c4f-52f9-4d74-aa06-be9f830bee8b-kube-api-access-klth4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:38.171998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:38.171964    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"
Nov 03 19:43:38.172197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:38.171960    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g" event={"ID":"bdde7c4f-52f9-4d74-aa06-be9f830bee8b","Type":"ContainerDied","Data":"a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028"}
Nov 03 19:43:38.172197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:38.172109    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86b71d4424560ce064776a499a8e9925c3cdedb2c15bc0ff762e878b648c028"
Nov 03 19:43:46.045681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:46.045643    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"]
Nov 03 19:43:46.092239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:46.092203    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hjwlr"]
Nov 03 19:43:47.932437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:47.932391    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a07180f7-868c-4d70-8ff4-c2671439c540" path="/var/lib/kubelet/pods/a07180f7-868c-4d70-8ff4-c2671439c540/volumes"
Nov 03 19:43:55.264064 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.264025    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"]
Nov 03 19:43:55.310434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.310397    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b652g"]
Nov 03 19:43:55.391150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.391114    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"]
Nov 03 19:43:55.391519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.391506    4944 cpu_manager.go:414] "containerMap: removing container" podUID="bdde7c4f-52f9-4d74-aa06-be9f830bee8b" containerName="revoke-keys"
Nov 03 19:43:55.391573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.391521    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdde7c4f-52f9-4d74-aa06-be9f830bee8b" containerName="revoke-keys"
Nov 03 19:43:55.391611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.391588    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdde7c4f-52f9-4d74-aa06-be9f830bee8b" containerName="revoke-keys"
Nov 03 19:43:55.395883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.395861    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.446950 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.446920    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:43:55.447164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.447102    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:43:55.453822 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.453779    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"]
Nov 03 19:43:55.473235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.473204    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:43:55.493241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.493220    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:43:55.572616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.572530    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.572616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.572573    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctjw\" (UniqueName: \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.572838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.572700    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.673513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.673475    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.673680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.673613    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.673680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.673639    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctjw\" (UniqueName: \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.674308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.674290    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.675808 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.675777    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.730828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.730781    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctjw\" (UniqueName: \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\") pod \"maas-api-revoke-sub-3d4156f1-cn5s2\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:55.933712 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:55.933633    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdde7c4f-52f9-4d74-aa06-be9f830bee8b" path="/var/lib/kubelet/pods/bdde7c4f-52f9-4d74-aa06-be9f830bee8b/volumes"
Nov 03 19:43:56.008837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:56.008777    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:56.376912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:56.376877    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"]
Nov 03 19:43:56.383429 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:43:56.383385    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2fcb7d_ebc8_4162_9463_08609c72954f.slice/crio-b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0 WatchSource:0}: Error finding container b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0: Status 404 returned error can't find the container with id b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0
Nov 03 19:43:57.268118 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:57.268085    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2" event={"ID":"aa2fcb7d-ebc8-4162-9463-08609c72954f","Type":"ContainerStarted","Data":"c1f960d5a040a15ed0f2010d46b6949c619ec00e4a80664f417c824f3da8d83c"}
Nov 03 19:43:57.268311 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:57.268161    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2" event={"ID":"aa2fcb7d-ebc8-4162-9463-08609c72954f","Type":"ContainerStarted","Data":"b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0"}
Nov 03 19:43:57.377291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:57.377227    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2" podStartSLOduration=2.377212433 podStartE2EDuration="2.377212433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:43:57.375302686 +0000 UTC m=+3535.652656282" watchObservedRunningTime="2026-11-03 19:43:57.377212433 +0000 UTC m=+3535.654566015"
Nov 03 19:43:58.274209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:58.274165    4944 generic.go:356] "Generic (PLEG): container finished" podID="aa2fcb7d-ebc8-4162-9463-08609c72954f" containerID="c1f960d5a040a15ed0f2010d46b6949c619ec00e4a80664f417c824f3da8d83c" exitCode=0
Nov 03 19:43:58.274423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:58.274206    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2" event={"ID":"aa2fcb7d-ebc8-4162-9463-08609c72954f","Type":"ContainerDied","Data":"c1f960d5a040a15ed0f2010d46b6949c619ec00e4a80664f417c824f3da8d83c"}
Nov 03 19:43:59.409722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.409697    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:43:59.512266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.512236    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\") pod \"aa2fcb7d-ebc8-4162-9463-08609c72954f\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") "
Nov 03 19:43:59.512459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.512369    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\" (UniqueName: \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\") pod \"aa2fcb7d-ebc8-4162-9463-08609c72954f\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") "
Nov 03 19:43:59.512459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.512403    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\") pod \"aa2fcb7d-ebc8-4162-9463-08609c72954f\" (UID: \"aa2fcb7d-ebc8-4162-9463-08609c72954f\") "
Nov 03 19:43:59.512845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.512819    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca" pod "aa2fcb7d-ebc8-4162-9463-08609c72954f" (UID: "aa2fcb7d-ebc8-4162-9463-08609c72954f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:43:59.514551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.514521    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw" pod "aa2fcb7d-ebc8-4162-9463-08609c72954f" (UID: "aa2fcb7d-ebc8-4162-9463-08609c72954f"). InnerVolumeSpecName "kube-api-access-dctjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:43:59.514737 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.514714    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca" pod "aa2fcb7d-ebc8-4162-9463-08609c72954f" (UID: "aa2fcb7d-ebc8-4162-9463-08609c72954f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:43:59.613348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.613261    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:59.613348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.613289    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctjw\" (UniqueName: \"kubernetes.io/projected/aa2fcb7d-ebc8-4162-9463-08609c72954f-kube-api-access-dctjw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:43:59.613348 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:43:59.613301    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2fcb7d-ebc8-4162-9463-08609c72954f-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:44:00.286137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:00.286102    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"
Nov 03 19:44:00.286318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:00.286136    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2" event={"ID":"aa2fcb7d-ebc8-4162-9463-08609c72954f","Type":"ContainerDied","Data":"b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0"}
Nov 03 19:44:00.286318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:00.286184    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0a5bc404e152d2c6f942252b02dd521245412511e3f7d5bce92ffc2377075f0"
Nov 03 19:44:05.223812 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:44:05.223759    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:44:05.841316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:05.841283    4944 scope.go:122] "RemoveContainer" containerID="d274fef006b8e3ebcb15afa4c9a2df2e0e502ea7c2400e6f107a3ba012823991"
Nov 03 19:44:05.849699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:05.849680    4944 scope.go:122] "RemoveContainer" containerID="a6c9574f0e0859964384affcfd10dedd336eb3a9ffc43130e11406c346b90c04"
Nov 03 19:44:11.927459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:11.927381    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:12.329976 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.329939    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:44:12.330335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.330322    4944 cpu_manager.go:414] "containerMap: removing container" podUID="aa2fcb7d-ebc8-4162-9463-08609c72954f" containerName="revoke-keys"
Nov 03 19:44:12.330382 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.330336    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2fcb7d-ebc8-4162-9463-08609c72954f" containerName="revoke-keys"
Nov 03 19:44:12.330418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.330406    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2fcb7d-ebc8-4162-9463-08609c72954f" containerName="revoke-keys"
Nov 03 19:44:12.335433 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.335414    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.388708 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.384555    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8ecb68-openshift-default-dockercfg-5wkvq\""
Nov 03 19:44:12.427979 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.427945    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428008    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428101    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428148 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428133    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthtr\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428152    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428306 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428200    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428330    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428387    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.428492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.428440    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.444093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.444065    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:44:12.529493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529459    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.529705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529497    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.529705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529521    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthtr\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.529705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529558    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.529705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529596    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.529705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529634    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529755    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529824    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.529914    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.530065    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530202 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.530107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.530368    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530457 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.530415    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.530626 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.530607    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.532086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.532054    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.532086 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.532058    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.535980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.535955    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:12.589641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.589579    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthtr\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.589641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.589580    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\") pod \"e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") " pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.649553 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.649518    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:12.891379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:12.891295    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:13.075350 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.075314    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:44:13.081300 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:44:13.081274    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34ae543_f59d_426a_80a8_93dd03758927.slice/crio-3d0df64ae97d511654dcb5f66f535112add9d6486e04c2905292ffa0f3e85eae WatchSource:0}: Error finding container 3d0df64ae97d511654dcb5f66f535112add9d6486e04c2905292ffa0f3e85eae: Status 404 returned error can't find the container with id 3d0df64ae97d511654dcb5f66f535112add9d6486e04c2905292ffa0f3e85eae
Nov 03 19:44:13.083566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.083532    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:13.083661 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.083628    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:13.083705 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.083665    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:13.350939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.350893    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerStarted","Data":"16e15264a8da0713ba9ba986ad6ebce1b748f9090e8b33941907377e2a88892e"}
Nov 03 19:44:13.351135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.350965    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerStarted","Data":"3d0df64ae97d511654dcb5f66f535112add9d6486e04c2905292ffa0f3e85eae"}
Nov 03 19:44:13.465920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.465867    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podStartSLOduration=1.4658532260000001 podStartE2EDuration="1.465853226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:13.464043955 +0000 UTC m=+3551.741397553" watchObservedRunningTime="2026-11-03 19:44:13.465853226 +0000 UTC m=+3551.743206809"
Nov 03 19:44:13.650013 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:13.649932    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:14.654762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:14.654727    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:15.359956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:15.359924    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:15.361109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:15.361088    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:16.462465 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:16.462428    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:16.747772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:16.747736    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:19.806599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:19.806562    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:20.054488 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.054453    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:20.101738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.101650    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:44:20.105089 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.105067    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.200207 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.200173    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8ecb68\""
Nov 03 19:44:20.201916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.201884    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-7bda81a1-dockercfg-hpt6c\""
Nov 03 19:44:20.203497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.203463    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvk66\" (UniqueName: \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.203636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.203521    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.204736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.204713    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:44:20.266050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.266019    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:44:20.269571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.269549    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.304460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.304425    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvk66\" (UniqueName: \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.304611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.304470    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.305204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.305185    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.328581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.328556    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:44:20.367601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.367526    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvk66\" (UniqueName: \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\") pod \"payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.405030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.404996    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.405199 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.405134    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q88l\" (UniqueName: \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.418299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.418272    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:20.506157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.506118    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q88l\" (UniqueName: \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.506339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.506208    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.506953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.506912    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.563606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.563575    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q88l\" (UniqueName: \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\") pod \"payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.583330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.583303    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:20.790212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.790178    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:44:20.796248 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:44:20.796218    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e7d510_3195_443a_be68_1ca54244ada5.slice/crio-d774220e2c2a3f2bbf15ea75d4ea545dbe94476f71dc38101f4c34193446478e WatchSource:0}: Error finding container d774220e2c2a3f2bbf15ea75d4ea545dbe94476f71dc38101f4c34193446478e: Status 404 returned error can't find the container with id d774220e2c2a3f2bbf15ea75d4ea545dbe94476f71dc38101f4c34193446478e
Nov 03 19:44:20.953388 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:20.953299    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:44:20.959106 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:44:20.959079    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e6220f_f75e_4de1_8896_e16409f4fb92.slice/crio-904b015e8b10f6e69484478f08938a31a2618e214c8ef75624e17300e2b352d9 WatchSource:0}: Error finding container 904b015e8b10f6e69484478f08938a31a2618e214c8ef75624e17300e2b352d9: Status 404 returned error can't find the container with id 904b015e8b10f6e69484478f08938a31a2618e214c8ef75624e17300e2b352d9
Nov 03 19:44:21.393327 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.393292    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" event={"ID":"e8e6220f-f75e-4de1-8896-e16409f4fb92","Type":"ContainerStarted","Data":"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"}
Nov 03 19:44:21.393545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.393351    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" event={"ID":"e8e6220f-f75e-4de1-8896-e16409f4fb92","Type":"ContainerStarted","Data":"904b015e8b10f6e69484478f08938a31a2618e214c8ef75624e17300e2b352d9"}
Nov 03 19:44:21.393545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.393380    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:21.394940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.394909    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" event={"ID":"29e7d510-3195-443a-be68-1ca54244ada5","Type":"ContainerStarted","Data":"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"}
Nov 03 19:44:21.395066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.394953    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" event={"ID":"29e7d510-3195-443a-be68-1ca54244ada5","Type":"ContainerStarted","Data":"d774220e2c2a3f2bbf15ea75d4ea545dbe94476f71dc38101f4c34193446478e"}
Nov 03 19:44:21.395066 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.394977    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:21.504119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.504066    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" podStartSLOduration=1.504052142 podStartE2EDuration="1.504052142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:21.502355224 +0000 UTC m=+3559.779708808" watchObservedRunningTime="2026-11-03 19:44:21.504052142 +0000 UTC m=+3559.781405725"
Nov 03 19:44:21.613091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:21.613033    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" podStartSLOduration=1.613020756 podStartE2EDuration="1.613020756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:21.611499974 +0000 UTC m=+3559.888853558" watchObservedRunningTime="2026-11-03 19:44:21.613020756 +0000 UTC m=+3559.890374338"
Nov 03 19:44:22.765020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:22.764985    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.011691 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:23.011651    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.264596 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:23.264564    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:23.523515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:23.523427    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:28.007688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:28.007649    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:32.403235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:32.403202    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:44:32.403744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:32.403283    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:44:32.997278 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:32.997242    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:35.244942 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:44:35.244898    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:44:37.461475 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:37.461443    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:38.021440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:38.021407    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:40.492880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.492842    4944 generic.go:356] "Generic (PLEG): container finished" podID="d34ae543-f59d-426a-80a8-93dd03758927" containerID="16e15264a8da0713ba9ba986ad6ebce1b748f9090e8b33941907377e2a88892e" exitCode=0
Nov 03 19:44:40.493349 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.492920    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerDied","Data":"16e15264a8da0713ba9ba986ad6ebce1b748f9090e8b33941907377e2a88892e"}
Nov 03 19:44:40.493402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.493367    4944 scope.go:122] "RemoveContainer" containerID="16e15264a8da0713ba9ba986ad6ebce1b748f9090e8b33941907377e2a88892e"
Nov 03 19:44:40.494621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.494597    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:40.494693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.494668    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:40.494732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:40.494703    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:44:41.235333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.235303    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:41.499673 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.499646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerStarted","Data":"197b70e2eb2d131505bfb8ae84b933b5ce837fd00aeb939d78ce5448a564f963"}
Nov 03 19:44:41.650592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.650556    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:41.652072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.652044    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 03 19:44:41.652221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.652092    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 03 19:44:41.891798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:41.891704    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.143095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:42.143003    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:42.650854 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:42.650818    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 03 19:44:42.651372 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:42.650880    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 03 19:44:43.044109 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.044069    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"]
Nov 03 19:44:43.090497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.090462    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-6g2mg"]
Nov 03 19:44:43.417545 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.417456    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:43.650345 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.650313    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 03 19:44:43.650520 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.650363    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 03 19:44:43.938971 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:43.938935    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="681f82f2-5308-456f-bcfa-0a932326c368" path="/var/lib/kubelet/pods/681f82f2-5308-456f-bcfa-0a932326c368/volumes"
Nov 03 19:44:44.649967 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:44.649925    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 03 19:44:44.650178 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:44.650001    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 03 19:44:45.360115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.360075    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:45.579855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.579820    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:44:45.583137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.583114    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.650590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.650487    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 03 19:44:45.650590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.650545    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 03 19:44:45.677249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677191    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.677458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677330    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.677458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677379    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2lbw\" (UniqueName: \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.677592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677473    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.677592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677513    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.677592 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.677552    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.685141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.685112    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8ecb68\"/\"kube-root-ca.crt\""
Nov 03 19:44:45.685141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.685119    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8ecb68\"/\"simulator-single-main-8ecb68-kserve-self-signed-certs\""
Nov 03 19:44:45.685352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.685125    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-8ecb68\"/\"default-dockercfg-8mgmj\""
Nov 03 19:44:45.685352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.685156    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-8ecb68\"/\"openshift-service-ca.crt\""
Nov 03 19:44:45.687430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.687408    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:44:45.778660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778622    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.778886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778685    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.778886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778732    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.778886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778770    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2lbw\" (UniqueName: \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.778886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778841    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.778886 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.778871    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.779172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.779107    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.779605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.779581    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.779732 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.779595    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.781865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.781843    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.783328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.783306    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.826814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.818422    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:45.837343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.837303    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2lbw\" (UniqueName: \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\") pod \"simulator-single-main-8ecb68-kserve-6469564d78-q48gl\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") " pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:45.894155 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:45.894114    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:46.293021 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.292986    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:44:46.299136 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:44:46.299110    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod507e5b39_ca82_4d86_94ca_9200c2c3002e.slice/crio-16e389f39392aec6c7f77603130d6d37c38195c125a3895b9952065641dce013 WatchSource:0}: Error finding container 16e389f39392aec6c7f77603130d6d37c38195c125a3895b9952065641dce013: Status 404 returned error can't find the container with id 16e389f39392aec6c7f77603130d6d37c38195c125a3895b9952065641dce013
Nov 03 19:44:46.555584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.555498    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" event={"ID":"507e5b39-ca82-4d86-94ca-9200c2c3002e","Type":"ContainerStarted","Data":"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"}
Nov 03 19:44:46.555584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.555548    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" event={"ID":"507e5b39-ca82-4d86-94ca-9200c2c3002e","Type":"ContainerStarted","Data":"16e389f39392aec6c7f77603130d6d37c38195c125a3895b9952065641dce013"}
Nov 03 19:44:46.654998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.654961    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:46.656098 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.656078    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:44:46.665307 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.665265    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" podStartSLOduration=1.665252779 podStartE2EDuration="1.665252779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:44:46.664767476 +0000 UTC m=+3584.942121255" watchObservedRunningTime="2026-11-03 19:44:46.665252779 +0000 UTC m=+3584.942606362"
Nov 03 19:44:46.895214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:46.895110    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:48.354508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:48.354460    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:53.497418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:53.497378    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:44:55.895279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:55.895239    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:55.908200 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:55.908169    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:56.620090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:56.620059    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:44:58.433401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:44:58.433366    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:00.163283 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.163250    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:00.168077 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.168056    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.217368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.217344    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:45:00.220876 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.220844    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.220960 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.220887    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpxcw\" (UniqueName: \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.263985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.263954    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:45:00.263985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.263960    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:45:00.266243 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.266222    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:00.321843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.321813    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.322020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.321848    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpxcw\" (UniqueName: \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.322427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.322408    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.379341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.379311    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpxcw\" (UniqueName: \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\") pod \"maas-api-key-cleanup-29895585-55cxz\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.481955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.481867    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:00.850056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:00.850017    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:00.856822 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:45:00.856765    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38191b17_6da0_479c_8e8d_fb5631582abb.slice/crio-5af214f59d530fffd08fde32aab6284ea9f896a9957b4cce9071237c432a75eb WatchSource:0}: Error finding container 5af214f59d530fffd08fde32aab6284ea9f896a9957b4cce9071237c432a75eb: Status 404 returned error can't find the container with id 5af214f59d530fffd08fde32aab6284ea9f896a9957b4cce9071237c432a75eb
Nov 03 19:45:01.638197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:01.638165    4944 generic.go:356] "Generic (PLEG): container finished" podID="38191b17-6da0-479c-8e8d-fb5631582abb" containerID="44cbafa95124dc8481b9f871a46db16dc2ac7601e6ed93f1668c8a1f48b90131" exitCode=60
Nov 03 19:45:01.638544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:01.638251    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerDied","Data":"44cbafa95124dc8481b9f871a46db16dc2ac7601e6ed93f1668c8a1f48b90131"}
Nov 03 19:45:01.638544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:01.638313    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerStarted","Data":"5af214f59d530fffd08fde32aab6284ea9f896a9957b4cce9071237c432a75eb"}
Nov 03 19:45:01.638544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:01.638485    4944 scope.go:122] "RemoveContainer" containerID="44cbafa95124dc8481b9f871a46db16dc2ac7601e6ed93f1668c8a1f48b90131"
Nov 03 19:45:02.497735 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.497699    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:45:02.498772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.498746    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:45:02.646762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.646727    4944 generic.go:356] "Generic (PLEG): container finished" podID="38191b17-6da0-479c-8e8d-fb5631582abb" containerID="2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587" exitCode=60
Nov 03 19:45:02.647225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.646830    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerDied","Data":"2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587"}
Nov 03 19:45:02.647225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.646939    4944 scope.go:122] "RemoveContainer" containerID="44cbafa95124dc8481b9f871a46db16dc2ac7601e6ed93f1668c8a1f48b90131"
Nov 03 19:45:02.647330 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:02.647285    4944 scope.go:122] "RemoveContainer" containerID="2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587"
Nov 03 19:45:02.647542 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:02.647519    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-55cxz_odh-ai-gateway-infra(38191b17-6da0-479c-8e8d-fb5631582abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" podUID="38191b17-6da0-479c-8e8d-fb5631582abb"
Nov 03 19:45:03.418016 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:03.417969    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:03.654268 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:03.654236    4944 scope.go:122] "RemoveContainer" containerID="2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587"
Nov 03 19:45:03.654756 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:03.654479    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-55cxz_odh-ai-gateway-infra(38191b17-6da0-479c-8e8d-fb5631582abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" podUID="38191b17-6da0-479c-8e8d-fb5631582abb"
Nov 03 19:45:05.223733 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:05.223698    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:45:05.873323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:05.873295    4944 scope.go:122] "RemoveContainer" containerID="3eb2e88b0e4da9fb462c1f45062f7d419d051af6c518028ed39342833e081f5b"
Nov 03 19:45:08.522365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:08.522331    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:12.926779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:12.926746    4944 scope.go:122] "RemoveContainer" containerID="2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587"
Nov 03 19:45:13.447508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:13.447465    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:13.713641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:13.713607    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerStarted","Data":"239e947831408454f1bfcd0977d9ea84a2d2a1bba77c8351fdd031fc9ebf05ba"}
Nov 03 19:45:13.822462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:13.822409    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" podStartSLOduration=13.822395902 podStartE2EDuration="13.822395902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:13.821218147 +0000 UTC m=+3612.098571729" watchObservedRunningTime="2026-11-03 19:45:13.822395902 +0000 UTC m=+3612.099749484"
Nov 03 19:45:14.720532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:14.720489    4944 generic.go:356] "Generic (PLEG): container finished" podID="38191b17-6da0-479c-8e8d-fb5631582abb" containerID="239e947831408454f1bfcd0977d9ea84a2d2a1bba77c8351fdd031fc9ebf05ba" exitCode=60
Nov 03 19:45:14.720978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:14.720545    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerDied","Data":"239e947831408454f1bfcd0977d9ea84a2d2a1bba77c8351fdd031fc9ebf05ba"}
Nov 03 19:45:14.720978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:14.720629    4944 scope.go:122] "RemoveContainer" containerID="2907aded23f0e1f8341e164c4c88249c6e61d4aacea3bfd0531793d1d8ec7587"
Nov 03 19:45:14.720978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:14.720951    4944 scope.go:122] "RemoveContainer" containerID="239e947831408454f1bfcd0977d9ea84a2d2a1bba77c8351fdd031fc9ebf05ba"
Nov 03 19:45:14.721240 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:14.721220    4944 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895585-55cxz_odh-ai-gateway-infra(38191b17-6da0-479c-8e8d-fb5631582abb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" podUID="38191b17-6da0-479c-8e8d-fb5631582abb"
Nov 03 19:45:14.837249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:14.837215    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:15.862753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:15.862729    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:15.973346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:15.973312    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\") pod \"38191b17-6da0-479c-8e8d-fb5631582abb\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") "
Nov 03 19:45:15.973346 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:15.973347    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\" (UniqueName: \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\") pod \"38191b17-6da0-479c-8e8d-fb5631582abb\" (UID: \"38191b17-6da0-479c-8e8d-fb5631582abb\") "
Nov 03 19:45:15.973728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:15.973704    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca" pod "38191b17-6da0-479c-8e8d-fb5631582abb" (UID: "38191b17-6da0-479c-8e8d-fb5631582abb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:15.975444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:15.975417    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw" pod "38191b17-6da0-479c-8e8d-fb5631582abb" (UID: "38191b17-6da0-479c-8e8d-fb5631582abb"). InnerVolumeSpecName "kube-api-access-fpxcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:16.044586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.044548    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"]
Nov 03 19:45:16.047622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.047596    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-tdsnw"]
Nov 03 19:45:16.074810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.074761    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38191b17-6da0-479c-8e8d-fb5631582abb-maas-api-service-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:16.074937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.074812    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpxcw\" (UniqueName: \"kubernetes.io/projected/38191b17-6da0-479c-8e8d-fb5631582abb-kube-api-access-fpxcw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:16.734726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.734697    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"
Nov 03 19:45:16.734937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.734698    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz" event={"ID":"38191b17-6da0-479c-8e8d-fb5631582abb","Type":"ContainerDied","Data":"5af214f59d530fffd08fde32aab6284ea9f896a9957b4cce9071237c432a75eb"}
Nov 03 19:45:16.734937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.734857    4944 scope.go:122] "RemoveContainer" containerID="239e947831408454f1bfcd0977d9ea84a2d2a1bba77c8351fdd031fc9ebf05ba"
Nov 03 19:45:16.887930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.887900    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:16.934448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:16.934405    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895585-55cxz"]
Nov 03 19:45:17.933641 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:17.933595    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" path="/var/lib/kubelet/pods/38191b17-6da0-479c-8e8d-fb5631582abb/volumes"
Nov 03 19:45:17.934277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:17.934255    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b089f3e1-8a27-4dbe-82cd-eb3e31885d9e" path="/var/lib/kubelet/pods/b089f3e1-8a27-4dbe-82cd-eb3e31885d9e/volumes"
Nov 03 19:45:18.485260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:18.485221    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:21.141633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.141599    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"]
Nov 03 19:45:21.142039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142003    4944 cpu_manager.go:414] "containerMap: removing container" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142014    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142044    4944 cpu_manager.go:414] "containerMap: removing container" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142050    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142109    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142120    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.142139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.142129    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:21.146603 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.146583    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.242078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.242047    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:21.242078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.242060    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:45:21.242299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.242099    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:45:21.242299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.242143    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:45:21.244654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.244632    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"]
Nov 03 19:45:21.332970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.332938    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.333169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.332983    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjcm\" (UniqueName: \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.333169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.333084    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.434611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.434527    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.434611 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.434576    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjcm\" (UniqueName: \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.434884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.434623    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.435363 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.435342    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.436859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.436827    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.492496 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.492469    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjcm\" (UniqueName: \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\") pod \"maas-api-revoke-sub-46e8a3dd-rmmn6\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:21.758804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:21.758762    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:22.127834 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:22.127739    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"]
Nov 03 19:45:22.134013 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:45:22.133976    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3af2ac_6c2f_4dcd_84ff_7324cfae77d7.slice/crio-50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d WatchSource:0}: Error finding container 50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d: Status 404 returned error can't find the container with id 50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d
Nov 03 19:45:22.767351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:22.767314    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6" event={"ID":"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7","Type":"ContainerStarted","Data":"49c999f2b4d4d048735350bc904d1388ae9a4ef2adf5b24c0fbb5c05de40608e"}
Nov 03 19:45:22.767782 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:22.767369    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6" event={"ID":"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7","Type":"ContainerStarted","Data":"50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d"}
Nov 03 19:45:22.876743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:22.876692    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6" podStartSLOduration=1.876677629 podStartE2EDuration="1.876677629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:22.875084767 +0000 UTC m=+3621.152438350" watchObservedRunningTime="2026-11-03 19:45:22.876677629 +0000 UTC m=+3621.154031190"
Nov 03 19:45:23.444864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:23.444826    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:23.774595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:23.774557    4944 generic.go:356] "Generic (PLEG): container finished" podID="ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" containerID="49c999f2b4d4d048735350bc904d1388ae9a4ef2adf5b24c0fbb5c05de40608e" exitCode=0
Nov 03 19:45:23.775085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:23.774644    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6" event={"ID":"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7","Type":"ContainerDied","Data":"49c999f2b4d4d048735350bc904d1388ae9a4ef2adf5b24c0fbb5c05de40608e"}
Nov 03 19:45:24.905305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.905281    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:24.967039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.967009    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\" (UniqueName: \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\") pod \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") "
Nov 03 19:45:24.967184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.967054    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\") pod \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") "
Nov 03 19:45:24.967184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.967076    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\") pod \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\" (UID: \"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7\") "
Nov 03 19:45:24.967497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.967472    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca" pod "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" (UID: "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:24.969156 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.969134    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca" pod "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" (UID: "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:24.969223 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:24.969201    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm" pod "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" (UID: "ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7"). InnerVolumeSpecName "kube-api-access-xdjcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:25.068900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.068816    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdjcm\" (UniqueName: \"kubernetes.io/projected/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-kube-api-access-xdjcm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.068900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.068848    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.068900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.068858    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:25.785831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.785781    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"
Nov 03 19:45:25.786012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.785781    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6" event={"ID":"ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7","Type":"ContainerDied","Data":"50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d"}
Nov 03 19:45:25.786012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:25.785920    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b57be05404246a6df0ef4e8e7e7d353ca7fe7d477c4e7c396d8279f7dfad0d"
Nov 03 19:45:28.043561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:28.043520    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"]
Nov 03 19:45:28.089959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:28.089921    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-dcjfh"]
Nov 03 19:45:28.372035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:28.371946    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:29.933100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:29.933063    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4df5f613-5139-48c7-8cfe-21de432ebb77" path="/var/lib/kubelet/pods/4df5f613-5139-48c7-8cfe-21de432ebb77/volumes"
Nov 03 19:45:33.151152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.151118    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.331303 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331267    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"]
Nov 03 19:45:33.331688 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331674    4944 cpu_manager.go:414] "containerMap: removing container" podUID="ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" containerName="revoke-keys"
Nov 03 19:45:33.331739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331688    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" containerName="revoke-keys"
Nov 03 19:45:33.331739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331705    4944 cpu_manager.go:414] "containerMap: removing container" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:33.331739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331711    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38191b17-6da0-479c-8e8d-fb5631582abb" containerName="cleanup"
Nov 03 19:45:33.331879 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.331808    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" containerName="revoke-keys"
Nov 03 19:45:33.335332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.335312    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.384429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.384403    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:45:33.404687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.404604    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:33.430828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.430782    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:45:33.430828 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.430811    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:45:33.431047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.430865    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:45:33.433137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.433120    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"]
Nov 03 19:45:33.444816 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.444767    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.444942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.444863    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.444942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.444910    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwssx\" (UniqueName: \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.545820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.545768    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.545989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.545835    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.545989 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.545867    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwssx\" (UniqueName: \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.546400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.546368    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.548183 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.548158    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.603557 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.603530    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwssx\" (UniqueName: \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:33.647748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:33.647716    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:34.015594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:34.015564    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"]
Nov 03 19:45:34.021340 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:45:34.021310    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd471be_c8de_4848_8689_ed51abc7b64b.slice/crio-73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65 WatchSource:0}: Error finding container 73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65: Status 404 returned error can't find the container with id 73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65
Nov 03 19:45:34.832680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:34.832630    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6" event={"ID":"bbd471be-c8de-4848-8689-ed51abc7b64b","Type":"ContainerStarted","Data":"e011b60b66df22c167071cd3e603a61bd9ee28abcf05931d51b060aba2cb1e4a"}
Nov 03 19:45:34.833107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:34.832704    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6" event={"ID":"bbd471be-c8de-4848-8689-ed51abc7b64b","Type":"ContainerStarted","Data":"73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65"}
Nov 03 19:45:34.943863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:34.943807    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6" podStartSLOduration=1.943779915 podStartE2EDuration="1.943779915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:45:34.94207572 +0000 UTC m=+3633.219429300" watchObservedRunningTime="2026-11-03 19:45:34.943779915 +0000 UTC m=+3633.221133497"
Nov 03 19:45:35.224360 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:35.224279    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:45:35.839590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:35.839557    4944 generic.go:356] "Generic (PLEG): container finished" podID="bbd471be-c8de-4848-8689-ed51abc7b64b" containerID="e011b60b66df22c167071cd3e603a61bd9ee28abcf05931d51b060aba2cb1e4a" exitCode=0
Nov 03 19:45:35.840095 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:35.839598    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6" event={"ID":"bbd471be-c8de-4848-8689-ed51abc7b64b","Type":"ContainerDied","Data":"e011b60b66df22c167071cd3e603a61bd9ee28abcf05931d51b060aba2cb1e4a"}
Nov 03 19:45:36.982383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:36.982359    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:37.083291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.083263    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\" (UniqueName: \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\") pod \"bbd471be-c8de-4848-8689-ed51abc7b64b\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") "
Nov 03 19:45:37.083484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.083308    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\") pod \"bbd471be-c8de-4848-8689-ed51abc7b64b\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") "
Nov 03 19:45:37.083484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.083348    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\") pod \"bbd471be-c8de-4848-8689-ed51abc7b64b\" (UID: \"bbd471be-c8de-4848-8689-ed51abc7b64b\") "
Nov 03 19:45:37.083704 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.083680    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca" pod "bbd471be-c8de-4848-8689-ed51abc7b64b" (UID: "bbd471be-c8de-4848-8689-ed51abc7b64b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:45:37.085440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.085410    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca" pod "bbd471be-c8de-4848-8689-ed51abc7b64b" (UID: "bbd471be-c8de-4848-8689-ed51abc7b64b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:37.085561 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.085493    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx" pod "bbd471be-c8de-4848-8689-ed51abc7b64b" (UID: "bbd471be-c8de-4848-8689-ed51abc7b64b"). InnerVolumeSpecName "kube-api-access-hwssx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:37.184945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.184865    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwssx\" (UniqueName: \"kubernetes.io/projected/bbd471be-c8de-4848-8689-ed51abc7b64b-kube-api-access-hwssx\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:37.184945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.184893    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:37.184945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.184903    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd471be-c8de-4848-8689-ed51abc7b64b-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:37.851160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.851127    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"
Nov 03 19:45:37.851347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.851161    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6" event={"ID":"bbd471be-c8de-4848-8689-ed51abc7b64b","Type":"ContainerDied","Data":"73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65"}
Nov 03 19:45:37.851347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.851224    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73cb384871b66c4d4182367e27ac08f506f8e4cdd94091ba5cbebd2fb920cf65"
Nov 03 19:45:37.858651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:37.858611    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.007378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:38.007341    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:45:38.189266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:38.189189    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:38.580531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:38.580499    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:43.592047 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:43.591967    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:44.969665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:44.969629    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:45:45.067351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:45.067312    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:45:45.414445 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:45.414396    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:45.669373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:45.669291    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:48.618982 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:48.618944    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:49.970884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:49.970840    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing" containerID="cri-o://4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef" gracePeriod=25
Nov 03 19:45:50.068497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:50.068452    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing" containerID="cri-o://800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8" gracePeriod=25
Nov 03 19:45:52.401119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:52.401079    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:45:52.401515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:52.401080    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:45:53.008955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.008886    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:53.027695 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.027625    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" podUID="507e5b39-ca82-4d86-94ca-9200c2c3002e" containerName="main" containerID="cri-o://c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9" gracePeriod=45
Nov 03 19:45:53.280803 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.280765    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:45:53.444470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444436    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\" (UniqueName: \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444524    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444560    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\" (UniqueName: \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444589    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444618    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444698    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\") pod \"507e5b39-ca82-4d86-94ca-9200c2c3002e\" (UID: \"507e5b39-ca82-4d86-94ca-9200c2c3002e\") "
Nov 03 19:45:53.444945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.444813    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:53.445356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.445047    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:53.445356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.445055    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.445356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.445066    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:53.447321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.447281    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:45:53.447432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.447361    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:45:53.447432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.447381    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw" pod "507e5b39-ca82-4d86-94ca-9200c2c3002e" (UID: "507e5b39-ca82-4d86-94ca-9200c2c3002e"). InnerVolumeSpecName "kube-api-access-c2lbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:45:53.546493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.546406    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/507e5b39-ca82-4d86-94ca-9200c2c3002e-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.546493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.546476    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.546493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.546489    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.546716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.546499    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/507e5b39-ca82-4d86-94ca-9200c2c3002e-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.546716 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.546511    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2lbw\" (UniqueName: \"kubernetes.io/projected/507e5b39-ca82-4d86-94ca-9200c2c3002e-kube-api-access-c2lbw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:45:53.938265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.938180    4944 generic.go:356] "Generic (PLEG): container finished" podID="507e5b39-ca82-4d86-94ca-9200c2c3002e" containerID="c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9" exitCode=0
Nov 03 19:45:53.938265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.938219    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"
Nov 03 19:45:53.938464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.938257    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" event={"ID":"507e5b39-ca82-4d86-94ca-9200c2c3002e","Type":"ContainerDied","Data":"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"}
Nov 03 19:45:53.938464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.938301    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl" event={"ID":"507e5b39-ca82-4d86-94ca-9200c2c3002e","Type":"ContainerDied","Data":"16e389f39392aec6c7f77603130d6d37c38195c125a3895b9952065641dce013"}
Nov 03 19:45:53.938464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.938321    4944 scope.go:122] "RemoveContainer" containerID="c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"
Nov 03 19:45:53.947447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.947431    4944 scope.go:122] "RemoveContainer" containerID="c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"
Nov 03 19:45:53.947739 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:45:53.947719    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9\": container with ID starting with c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9 not found: ID does not exist" containerID="c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"
Nov 03 19:45:53.947804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:53.947747    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9"} err="failed to get container status \"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9\": rpc error: code = NotFound desc = could not find container \"c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9\": container with ID starting with c62bce3e3cb0613927db15620d30f5d0f85d5904643aa8c0902a3799c5bca8b9 not found: ID does not exist"
Nov 03 19:45:54.092186 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:54.092147    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:45:54.138643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:54.138605    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-8ecb68/simulator-single-main-8ecb68-kserve-6469564d78-q48gl"]
Nov 03 19:45:54.614028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:54.613989    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:54.899943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:54.899860    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.129920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:55.129885    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"]
Nov 03 19:45:55.163810 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:55.163725    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8ecb68-4vsn6"]
Nov 03 19:45:55.688817 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:55.688757    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:45:55.933005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:55.932964    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507e5b39-ca82-4d86-94ca-9200c2c3002e" path="/var/lib/kubelet/pods/507e5b39-ca82-4d86-94ca-9200c2c3002e/volumes"
Nov 03 19:45:55.933479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:45:55.933458    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd471be-c8de-4848-8689-ed51abc7b64b" path="/var/lib/kubelet/pods/bbd471be-c8de-4848-8689-ed51abc7b64b/volumes"
Nov 03 19:46:01.629955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:01.629924    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:01.869043 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:01.869009    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:46:01.869244 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:01.869221    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy" containerID="cri-o://197b70e2eb2d131505bfb8ae84b933b5ce837fd00aeb939d78ce5448a564f963" gracePeriod=30
Nov 03 19:46:01.887072 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:01.887008    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:02.134636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:02.134597    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:02.401419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:02.401380    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:46:02.401419 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:02.401414    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:46:03.044720 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.044678    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"]
Nov 03 19:46:03.049830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.049800    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-bj2cj"]
Nov 03 19:46:03.503032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.502998    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"]
Nov 03 19:46:03.505627 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.505598    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cn5s2"]
Nov 03 19:46:03.632102 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632069    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"]
Nov 03 19:46:03.632552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632532    4944 cpu_manager.go:414] "containerMap: removing container" podUID="507e5b39-ca82-4d86-94ca-9200c2c3002e" containerName="main"
Nov 03 19:46:03.632660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632555    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507e5b39-ca82-4d86-94ca-9200c2c3002e" containerName="main"
Nov 03 19:46:03.632660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632570    4944 cpu_manager.go:414] "containerMap: removing container" podUID="bbd471be-c8de-4848-8689-ed51abc7b64b" containerName="revoke-keys"
Nov 03 19:46:03.632660 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632578    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd471be-c8de-4848-8689-ed51abc7b64b" containerName="revoke-keys"
Nov 03 19:46:03.632889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632694    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="507e5b39-ca82-4d86-94ca-9200c2c3002e" containerName="main"
Nov 03 19:46:03.632889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.632711    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd471be-c8de-4848-8689-ed51abc7b64b" containerName="revoke-keys"
Nov 03 19:46:03.639577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.639538    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.694509 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.694480    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:46:03.694669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.694568    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:46:03.696711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.696690    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"]
Nov 03 19:46:03.699460 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.699432    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:46:03.699593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.699528    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:46:03.743625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.743593    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qq8m\" (UniqueName: \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.743774 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.743639    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.743846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.743769    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.844651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.844547    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.844651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.844649    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qq8m\" (UniqueName: \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.844965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.844676    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.845387 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.845365    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.847025 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.847000    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.902361 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.902322    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qq8m\" (UniqueName: \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\") pod \"maas-api-revoke-sub-3d4156f1-rbv46\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:03.932702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.932672    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2fcb7d-ebc8-4162-9463-08609c72954f" path="/var/lib/kubelet/pods/aa2fcb7d-ebc8-4162-9463-08609c72954f/volumes"
Nov 03 19:46:03.933125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.933110    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32cdd9c-19bb-4cc9-bba7-a585f5a0819d" path="/var/lib/kubelet/pods/d32cdd9c-19bb-4cc9-bba7-a585f5a0819d/volumes"
Nov 03 19:46:03.952883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:03.952849    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:04.327740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:04.327705    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"]
Nov 03 19:46:04.333980 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:46:04.333942    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5fd0b7_557c_4703_8886_f43206fe39cd.slice/crio-64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1 WatchSource:0}: Error finding container 64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1: Status 404 returned error can't find the container with id 64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1
Nov 03 19:46:04.997315 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:04.997277    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46" event={"ID":"3f5fd0b7-557c-4703-8886-f43206fe39cd","Type":"ContainerStarted","Data":"26c7a8a7bd8250a34ec3732b8206d9eb2ecbe086199d71228dad3801604fde02"}
Nov 03 19:46:04.997497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:04.997343    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46" event={"ID":"3f5fd0b7-557c-4703-8886-f43206fe39cd","Type":"ContainerStarted","Data":"64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1"}
Nov 03 19:46:05.106224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:05.106173    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46" podStartSLOduration=2.106158387 podStartE2EDuration="2.106158387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:46:05.105598328 +0000 UTC m=+3663.382951914" watchObservedRunningTime="2026-11-03 19:46:05.106158387 +0000 UTC m=+3663.383511977"
Nov 03 19:46:05.224176 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:46:05.224135    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:46:05.893835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:05.893805    4944 scope.go:122] "RemoveContainer" containerID="ce5f69d1566c60be81903b20428f05ef85a1d04e63b9f0fbbd3b2548c5572687"
Nov 03 19:46:06.004227 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:06.004195    4944 generic.go:356] "Generic (PLEG): container finished" podID="3f5fd0b7-557c-4703-8886-f43206fe39cd" containerID="26c7a8a7bd8250a34ec3732b8206d9eb2ecbe086199d71228dad3801604fde02" exitCode=0
Nov 03 19:46:06.004404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:06.004288    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46" event={"ID":"3f5fd0b7-557c-4703-8886-f43206fe39cd","Type":"ContainerDied","Data":"26c7a8a7bd8250a34ec3732b8206d9eb2ecbe086199d71228dad3801604fde02"}
Nov 03 19:46:07.012285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.012246    4944 generic.go:356] "Generic (PLEG): container finished" podID="d34ae543-f59d-426a-80a8-93dd03758927" containerID="197b70e2eb2d131505bfb8ae84b933b5ce837fd00aeb939d78ce5448a564f963" exitCode=0
Nov 03 19:46:07.012672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.012303    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerDied","Data":"197b70e2eb2d131505bfb8ae84b933b5ce837fd00aeb939d78ce5448a564f963"}
Nov 03 19:46:07.012672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.012365    4944 scope.go:122] "RemoveContainer" containerID="16e15264a8da0713ba9ba986ad6ebce1b748f9090e8b33941907377e2a88892e"
Nov 03 19:46:07.138110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.138039    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:46:07.169402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.169376    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:07.175821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175785    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.175895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175849    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.175895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175876    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.175987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175903    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.175987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175957    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.175987 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.175983    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\" (UniqueName: \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\") pod \"3f5fd0b7-557c-4703-8886-f43206fe39cd\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") "
Nov 03 19:46:07.176149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176021    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\") pod \"3f5fd0b7-557c-4703-8886-f43206fe39cd\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") "
Nov 03 19:46:07.176149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176050    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.176149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176078    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\") pod \"3f5fd0b7-557c-4703-8886-f43206fe39cd\" (UID: \"3f5fd0b7-557c-4703-8886-f43206fe39cd\") "
Nov 03 19:46:07.176149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176121    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.176149 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176132    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:46:07.176432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176161    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.176432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176193    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\") pod \"d34ae543-f59d-426a-80a8-93dd03758927\" (UID: \"d34ae543-f59d-426a-80a8-93dd03758927\") "
Nov 03 19:46:07.176432 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176419    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:46:07.176605 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176461    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca" pod "3f5fd0b7-557c-4703-8886-f43206fe39cd" (UID: "3f5fd0b7-557c-4703-8886-f43206fe39cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:07.176734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176677    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.176734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176702    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.176734 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.176718    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.177503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.177000    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:46:07.177503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.177079    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:46:07.177503 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.177158    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:07.178423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.178397    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:46:07.178629 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.178610    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:07.178872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.178851    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca" pod "3f5fd0b7-557c-4703-8886-f43206fe39cd" (UID: "3f5fd0b7-557c-4703-8886-f43206fe39cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:46:07.178970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.178951    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "kube-api-access-vthtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:07.179032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.178992    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy" pod "d34ae543-f59d-426a-80a8-93dd03758927" (UID: "d34ae543-f59d-426a-80a8-93dd03758927"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:46:07.179229 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.179210    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m" pod "3f5fd0b7-557c-4703-8886-f43206fe39cd" (UID: "3f5fd0b7-557c-4703-8886-f43206fe39cd"). InnerVolumeSpecName "kube-api-access-5qq8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:07.278175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278143    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278173    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278183    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d34ae543-f59d-426a-80a8-93dd03758927-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278192    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d34ae543-f59d-426a-80a8-93dd03758927-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278201    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278209    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d34ae543-f59d-426a-80a8-93dd03758927-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278217    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthtr\" (UniqueName: \"kubernetes.io/projected/d34ae543-f59d-426a-80a8-93dd03758927-kube-api-access-vthtr\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278228    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qq8m\" (UniqueName: \"kubernetes.io/projected/3f5fd0b7-557c-4703-8886-f43206fe39cd-kube-api-access-5qq8m\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:07.278378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:07.278236    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5fd0b7-557c-4703-8886-f43206fe39cd-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:08.019185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.019151    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"
Nov 03 19:46:08.019723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.019182    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f" event={"ID":"d34ae543-f59d-426a-80a8-93dd03758927","Type":"ContainerDied","Data":"3d0df64ae97d511654dcb5f66f535112add9d6486e04c2905292ffa0f3e85eae"}
Nov 03 19:46:08.019723 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.019254    4944 scope.go:122] "RemoveContainer" containerID="197b70e2eb2d131505bfb8ae84b933b5ce837fd00aeb939d78ce5448a564f963"
Nov 03 19:46:08.020610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.020522    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"
Nov 03 19:46:08.020610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.020533    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46" event={"ID":"3f5fd0b7-557c-4703-8886-f43206fe39cd","Type":"ContainerDied","Data":"64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1"}
Nov 03 19:46:08.020610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.020569    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64b6b42aa6c64b414bde34b26a131acc6f79bf8772a050614a8f8b71c74308b1"
Nov 03 19:46:08.222107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.222019    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:46:08.228554 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:08.228526    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8ecb68-openshift-default-7dbf896cb9sjb4f"]
Nov 03 19:46:09.932065 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:09.932030    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d34ae543-f59d-426a-80a8-93dd03758927" path="/var/lib/kubelet/pods/d34ae543-f59d-426a-80a8-93dd03758927/volumes"
Nov 03 19:46:10.229479 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.229447    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:46:10.305650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.305623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\" (UniqueName: \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\") pod \"e8e6220f-f75e-4de1-8896-e16409f4fb92\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") "
Nov 03 19:46:10.305767 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.305662    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\") pod \"e8e6220f-f75e-4de1-8896-e16409f4fb92\" (UID: \"e8e6220f-f75e-4de1-8896-e16409f4fb92\") "
Nov 03 19:46:10.306103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.306083    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume" pod "e8e6220f-f75e-4de1-8896-e16409f4fb92" (UID: "e8e6220f-f75e-4de1-8896-e16409f4fb92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:10.307769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.307743    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l" pod "e8e6220f-f75e-4de1-8896-e16409f4fb92" (UID: "e8e6220f-f75e-4de1-8896-e16409f4fb92"). InnerVolumeSpecName "kube-api-access-4q88l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:10.313123 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.313101    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:46:10.406486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.406451    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\") pod \"29e7d510-3195-443a-be68-1ca54244ada5\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") "
Nov 03 19:46:10.406486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.406489    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\" (UniqueName: \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\") pod \"29e7d510-3195-443a-be68-1ca54244ada5\" (UID: \"29e7d510-3195-443a-be68-1ca54244ada5\") "
Nov 03 19:46:10.406707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.406683    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q88l\" (UniqueName: \"kubernetes.io/projected/e8e6220f-f75e-4de1-8896-e16409f4fb92-kube-api-access-4q88l\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:10.406707 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.406694    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8e6220f-f75e-4de1-8896-e16409f4fb92-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:10.406868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.406846    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume" pod "29e7d510-3195-443a-be68-1ca54244ada5" (UID: "29e7d510-3195-443a-be68-1ca54244ada5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:46:10.408527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.408508    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66" pod "29e7d510-3195-443a-be68-1ca54244ada5" (UID: "29e7d510-3195-443a-be68-1ca54244ada5"). InnerVolumeSpecName "kube-api-access-hvk66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:46:10.507916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.507879    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29e7d510-3195-443a-be68-1ca54244ada5-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:10.507916 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.507910    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvk66\" (UniqueName: \"kubernetes.io/projected/29e7d510-3195-443a-be68-1ca54244ada5-kube-api-access-hvk66\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:46:10.934837 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:10.934733    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:11.036797 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.036752    4944 generic.go:356] "Generic (PLEG): container finished" podID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerID="4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef" exitCode=0
Nov 03 19:46:11.037001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.036822    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"
Nov 03 19:46:11.037001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.036823    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" event={"ID":"e8e6220f-f75e-4de1-8896-e16409f4fb92","Type":"ContainerDied","Data":"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"}
Nov 03 19:46:11.037001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.036949    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn" event={"ID":"e8e6220f-f75e-4de1-8896-e16409f4fb92","Type":"ContainerDied","Data":"904b015e8b10f6e69484478f08938a31a2618e214c8ef75624e17300e2b352d9"}
Nov 03 19:46:11.037001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.036982    4944 scope.go:122] "RemoveContainer" containerID="4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"
Nov 03 19:46:11.038184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.038164    4944 generic.go:356] "Generic (PLEG): container finished" podID="29e7d510-3195-443a-be68-1ca54244ada5" containerID="800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8" exitCode=0
Nov 03 19:46:11.038260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.038194    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" event={"ID":"29e7d510-3195-443a-be68-1ca54244ada5","Type":"ContainerDied","Data":"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"}
Nov 03 19:46:11.038260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.038213    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"
Nov 03 19:46:11.038260 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.038229    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t" event={"ID":"29e7d510-3195-443a-be68-1ca54244ada5","Type":"ContainerDied","Data":"d774220e2c2a3f2bbf15ea75d4ea545dbe94476f71dc38101f4c34193446478e"}
Nov 03 19:46:11.045931 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.045812    4944 scope.go:122] "RemoveContainer" containerID="4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"
Nov 03 19:46:11.046168 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:46:11.046139    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef\": container with ID starting with 4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef not found: ID does not exist" containerID="4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"
Nov 03 19:46:11.046253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.046166    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef"} err="failed to get container status \"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef\": rpc error: code = NotFound desc = could not find container \"4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef\": container with ID starting with 4bc0a6c929efeefd31a434b03984e631deb8401912ae9f8934990f7e8411dbef not found: ID does not exist"
Nov 03 19:46:11.046253 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.046187    4944 scope.go:122] "RemoveContainer" containerID="800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"
Nov 03 19:46:11.054197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.054180    4944 scope.go:122] "RemoveContainer" containerID="800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"
Nov 03 19:46:11.054486 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:46:11.054464    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8\": container with ID starting with 800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8 not found: ID does not exist" containerID="800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"
Nov 03 19:46:11.054574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.054557    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8"} err="failed to get container status \"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8\": rpc error: code = NotFound desc = could not find container \"800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8\": container with ID starting with 800c101a23494339b0bd72aefd50298261c11ad85f70ed5dd718bf13484c92f8 not found: ID does not exist"
Nov 03 19:46:11.192906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.192820    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:46:11.239197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.239164    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8ecb68-6fdf68vn5t"]
Nov 03 19:46:11.306476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.306439    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:46:11.310234 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.310206    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-8ecb68-797d5f976nh4pn"]
Nov 03 19:46:11.932368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.932334    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e7d510-3195-443a-be68-1ca54244ada5" path="/var/lib/kubelet/pods/29e7d510-3195-443a-be68-1ca54244ada5/volumes"
Nov 03 19:46:11.932724 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:11.932710    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" path="/var/lib/kubelet/pods/e8e6220f-f75e-4de1-8896-e16409f4fb92/volumes"
Nov 03 19:46:14.044196 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:14.044161    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"]
Nov 03 19:46:14.090312 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:14.090276    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-z6vsd"]
Nov 03 19:46:14.995552 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:14.995511    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.397912 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:15.397768    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:15.932880 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:15.932848    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0dbea4-d04b-4559-83e9-4641dd2171d6" path="/var/lib/kubelet/pods/0d0dbea4-d04b-4559-83e9-4641dd2171d6/volumes"
Nov 03 19:46:21.066380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:21.066348    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:35.223592 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:46:35.223556    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:46:38.186846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:38.186811    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:38.437020 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:38.436935    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:46:49.043759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:49.043723    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"]
Nov 03 19:46:49.093175 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:49.093139    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-7wl7n"]
Nov 03 19:46:49.932811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:46:49.932758    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7b79391-20f7-4e55-883b-5606e457917a" path="/var/lib/kubelet/pods/a7b79391-20f7-4e55-883b-5606e457917a/volumes"
Nov 03 19:47:05.223678 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:47:05.223641    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:47:05.926868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:05.926833    4944 scope.go:122] "RemoveContainer" containerID="88a9b197bc12941384db96cfc1a6fb1351f53ca0ab92fcf251b466a84b4d1b02"
Nov 03 19:47:05.935600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:05.935579    4944 scope.go:122] "RemoveContainer" containerID="0a7370a6c08e1db0c9a82c16ddd97bad93822ea61e24080ae91299de1f1bf96c"
Nov 03 19:47:05.943205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:05.943186    4944 scope.go:122] "RemoveContainer" containerID="76956c4d098b569ff85cc7c5a5665895f53bdaf6db75fe73db448f1c170e9373"
Nov 03 19:47:17.043954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:17.043866    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"]
Nov 03 19:47:17.090480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:17.090443    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-m9whs"]
Nov 03 19:47:17.931884 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:17.931850    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5996061c-2522-4b68-9ba5-0a3bde033e9d" path="/var/lib/kubelet/pods/5996061c-2522-4b68-9ba5-0a3bde033e9d/volumes"
Nov 03 19:47:35.224014 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:47:35.223977    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:47:38.537633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:38.537598    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:42.073470 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:42.073435    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:47:48.044029 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:48.043991    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"]
Nov 03 19:47:48.090526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:48.090486    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-s9b6s"]
Nov 03 19:47:49.932763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:49.932727    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8774b553-0ec0-43a3-961a-80ddc3d4e94f" path="/var/lib/kubelet/pods/8774b553-0ec0-43a3-961a-80ddc3d4e94f/volumes"
Nov 03 19:47:54.423824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.423749    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"]
Nov 03 19:47:54.424514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424496    4944 cpu_manager.go:414] "containerMap: removing container" podUID="3f5fd0b7-557c-4703-8886-f43206fe39cd" containerName="revoke-keys"
Nov 03 19:47:54.424573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424514    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5fd0b7-557c-4703-8886-f43206fe39cd" containerName="revoke-keys"
Nov 03 19:47:54.424573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424560    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:47:54.424573 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424566    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:47:54.424759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424584    4944 cpu_manager.go:414] "containerMap: removing container" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing"
Nov 03 19:47:54.424759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424589    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing"
Nov 03 19:47:54.424759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424600    4944 cpu_manager.go:414] "containerMap: removing container" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing"
Nov 03 19:47:54.424759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424606    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing"
Nov 03 19:47:54.425001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424815    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5fd0b7-557c-4703-8886-f43206fe39cd" containerName="revoke-keys"
Nov 03 19:47:54.425001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424831    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:47:54.425001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424840    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:47:54.425001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424847    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e7d510-3195-443a-be68-1ca54244ada5" containerName="payload-pre-processing"
Nov 03 19:47:54.425001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.424854    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e6220f-f75e-4de1-8896-e16409f4fb92" containerName="payload-processing"
Nov 03 19:47:54.429233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.429213    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.478161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.478120    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:47:54.478337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.478222    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:47:54.524571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.524540    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:47:54.524748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.524572    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:47:54.526760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.526740    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"]
Nov 03 19:47:54.590994 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.590961    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.591162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.591025    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7llcz\" (UniqueName: \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.591162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.591133    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.692157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.692069    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.692324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.692160    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.692324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.692217    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7llcz\" (UniqueName: \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.692874 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.692854    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.694542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.694523    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:54.767745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:54.767708    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7llcz\" (UniqueName: \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\") pod \"maas-api-revoke-sub-d576e68b-s6745\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:55.042094 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:55.042057    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:55.407231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:55.407151    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"]
Nov 03 19:47:55.413882 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:47:55.413851    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b90d1e_c262_4a44_b0fa_951ca63a70a0.slice/crio-fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0 WatchSource:0}: Error finding container fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0: Status 404 returned error can't find the container with id fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0
Nov 03 19:47:55.566039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:55.565996    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745" event={"ID":"d6b90d1e-c262-4a44-b0fa-951ca63a70a0","Type":"ContainerStarted","Data":"fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0"}
Nov 03 19:47:56.572609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:56.572573    4944 generic.go:356] "Generic (PLEG): container finished" podID="d6b90d1e-c262-4a44-b0fa-951ca63a70a0" containerID="c25dc3a3406fabad1740d61e0ffe97dd29ae6aa5a3f3676c477bec64775db390" exitCode=0
Nov 03 19:47:56.573026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:56.572662    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745" event={"ID":"d6b90d1e-c262-4a44-b0fa-951ca63a70a0","Type":"ContainerDied","Data":"c25dc3a3406fabad1740d61e0ffe97dd29ae6aa5a3f3676c477bec64775db390"}
Nov 03 19:47:57.704418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.704386    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:57.823092 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.823060    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\") pod \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") "
Nov 03 19:47:57.823280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.823138    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\") pod \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") "
Nov 03 19:47:57.823280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.823253    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\" (UniqueName: \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\") pod \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\" (UID: \"d6b90d1e-c262-4a44-b0fa-951ca63a70a0\") "
Nov 03 19:47:57.823417 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.823386    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca" pod "d6b90d1e-c262-4a44-b0fa-951ca63a70a0" (UID: "d6b90d1e-c262-4a44-b0fa-951ca63a70a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:47:57.823526 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.823507    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:57.825515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.825487    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca" pod "d6b90d1e-c262-4a44-b0fa-951ca63a70a0" (UID: "d6b90d1e-c262-4a44-b0fa-951ca63a70a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:47:57.825667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.825647    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz" pod "d6b90d1e-c262-4a44-b0fa-951ca63a70a0" (UID: "d6b90d1e-c262-4a44-b0fa-951ca63a70a0"). InnerVolumeSpecName "kube-api-access-7llcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:47:57.924318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.924278    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7llcz\" (UniqueName: \"kubernetes.io/projected/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-kube-api-access-7llcz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:57.924318 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:57.924313    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6b90d1e-c262-4a44-b0fa-951ca63a70a0-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:47:58.583004 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:58.582970    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"
Nov 03 19:47:58.583184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:58.583011    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745" event={"ID":"d6b90d1e-c262-4a44-b0fa-951ca63a70a0","Type":"ContainerDied","Data":"fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0"}
Nov 03 19:47:58.583184 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:47:58.583059    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb2a62a8e1c879be770a995a19b4794c9fcfc0d7684efaca517ea01b7636ffc0"
Nov 03 19:48:05.224298 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:48:05.224261    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:48:05.978633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:05.978599    4944 scope.go:122] "RemoveContainer" containerID="662db66cd39cbf57085f5cdb1e47af81303056f52b059ab03666e68179cefa43"
Nov 03 19:48:05.986762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:05.986735    4944 scope.go:122] "RemoveContainer" containerID="87ecc6e4af31bb55ec130c1efea658b284af01e359f87e126ef39fb61bd6bb2b"
Nov 03 19:48:09.048473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:09.048432    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"]
Nov 03 19:48:09.053835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:09.053781    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4k2fw"]
Nov 03 19:48:09.932527 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:09.932493    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85c1ba47-4490-46cd-b5b2-8081207554c9" path="/var/lib/kubelet/pods/85c1ba47-4490-46cd-b5b2-8081207554c9/volumes"
Nov 03 19:48:16.083600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.083560    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"]
Nov 03 19:48:16.084040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.083970    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d6b90d1e-c262-4a44-b0fa-951ca63a70a0" containerName="revoke-keys"
Nov 03 19:48:16.084040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.083981    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b90d1e-c262-4a44-b0fa-951ca63a70a0" containerName="revoke-keys"
Nov 03 19:48:16.084040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.083995    4944 cpu_manager.go:414] "containerMap: removing container" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:48:16.084040 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.084000    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d34ae543-f59d-426a-80a8-93dd03758927" containerName="istio-proxy"
Nov 03 19:48:16.084191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.084108    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b90d1e-c262-4a44-b0fa-951ca63a70a0" containerName="revoke-keys"
Nov 03 19:48:16.088448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.088428    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.106238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.106211    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.106367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.106268    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.106418 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.106366    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596bv\" (UniqueName: \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.137236 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.137212    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:48:16.137352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.137286    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:48:16.137352 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.137308    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:48:16.183709 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.183680    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:48:16.185636 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.185615    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"]
Nov 03 19:48:16.207249 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.207223    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.207343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.207274    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.207343 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.207314    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596bv\" (UniqueName: \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.208034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.208016    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.209628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.209611    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.264584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.264550    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596bv\" (UniqueName: \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\") pod \"maas-api-revoke-sub-d2bc93bc-qwvxg\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.400745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.400660    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:16.768356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:16.768323    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"]
Nov 03 19:48:16.774507 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:48:16.774479    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fcf900_e509_483c_a534_b9a45e46f5d1.slice/crio-48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0 WatchSource:0}: Error finding container 48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0: Status 404 returned error can't find the container with id 48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0
Nov 03 19:48:17.702529 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:17.702491    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg" event={"ID":"93fcf900-e509-483c-a534-b9a45e46f5d1","Type":"ContainerStarted","Data":"2bc483d310414907d1d52e17a789e20c056d56c738e6626a18c9a10c373789e2"}
Nov 03 19:48:17.702939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:17.702547    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg" event={"ID":"93fcf900-e509-483c-a534-b9a45e46f5d1","Type":"ContainerStarted","Data":"48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0"}
Nov 03 19:48:17.811099 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:17.811055    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg" podStartSLOduration=1.81104127 podStartE2EDuration="1.81104127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:17.810361604 +0000 UTC m=+3796.087715188" watchObservedRunningTime="2026-11-03 19:48:17.81104127 +0000 UTC m=+3796.088394853"
Nov 03 19:48:18.709254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:18.709219    4944 generic.go:356] "Generic (PLEG): container finished" podID="93fcf900-e509-483c-a534-b9a45e46f5d1" containerID="2bc483d310414907d1d52e17a789e20c056d56c738e6626a18c9a10c373789e2" exitCode=0
Nov 03 19:48:18.709665 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:18.709298    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg" event={"ID":"93fcf900-e509-483c-a534-b9a45e46f5d1","Type":"ContainerDied","Data":"2bc483d310414907d1d52e17a789e20c056d56c738e6626a18c9a10c373789e2"}
Nov 03 19:48:19.844277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.844251    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:19.941781 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.941745    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\" (UniqueName: \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\") pod \"93fcf900-e509-483c-a534-b9a45e46f5d1\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") "
Nov 03 19:48:19.941983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.941870    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\") pod \"93fcf900-e509-483c-a534-b9a45e46f5d1\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") "
Nov 03 19:48:19.941983 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.941977    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\") pod \"93fcf900-e509-483c-a534-b9a45e46f5d1\" (UID: \"93fcf900-e509-483c-a534-b9a45e46f5d1\") "
Nov 03 19:48:19.942323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.942299    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca" pod "93fcf900-e509-483c-a534-b9a45e46f5d1" (UID: "93fcf900-e509-483c-a534-b9a45e46f5d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:48:19.943889 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.943858    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv" pod "93fcf900-e509-483c-a534-b9a45e46f5d1" (UID: "93fcf900-e509-483c-a534-b9a45e46f5d1"). InnerVolumeSpecName "kube-api-access-596bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:48:19.943986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:19.943914    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca" pod "93fcf900-e509-483c-a534-b9a45e46f5d1" (UID: "93fcf900-e509-483c-a534-b9a45e46f5d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:48:20.042939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.042897    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:20.042939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.042930    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596bv\" (UniqueName: \"kubernetes.io/projected/93fcf900-e509-483c-a534-b9a45e46f5d1-kube-api-access-596bv\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:20.043140 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.042960    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93fcf900-e509-483c-a534-b9a45e46f5d1-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:48:20.721045 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.721017    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"
Nov 03 19:48:20.721235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.721056    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg" event={"ID":"93fcf900-e509-483c-a534-b9a45e46f5d1","Type":"ContainerDied","Data":"48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0"}
Nov 03 19:48:20.721235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:20.721107    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48bfeb736271fda6aa8caa199ec518ac94422452bb10fcf70304442128349dd0"
Nov 03 19:48:28.324001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.323958    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.700100 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.700006    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:48:28.700499 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.700479    4944 cpu_manager.go:414] "containerMap: removing container" podUID="93fcf900-e509-483c-a534-b9a45e46f5d1" containerName="revoke-keys"
Nov 03 19:48:28.700593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.700501    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93fcf900-e509-483c-a534-b9a45e46f5d1" containerName="revoke-keys"
Nov 03 19:48:28.700662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.700600    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="93fcf900-e509-483c-a534-b9a45e46f5d1" containerName="revoke-keys"
Nov 03 19:48:28.703811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.703777    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.753137 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.753091    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ccf773-openshift-default-dockercfg-s6v26\""
Nov 03 19:48:28.807153 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.807119    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:48:28.828970 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.828938    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq64\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829047    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829075    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829126    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829196    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829254 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829235    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829253    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829285    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.829399 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.829364    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.901710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.901677    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:28.930088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930051    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930093    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930114    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930143    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930281    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930343    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq64\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930434    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930519 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930468    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930500    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930763 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930703    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930861    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930945 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930921    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.930996 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.930959    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.931034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.931012    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.932693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.932665    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.932693 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.932687    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:28.998669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:28.998642    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:29.029012 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.028986    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq64\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\") pod \"e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") " pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:29.201157 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.201119    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:29.316379 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.316286    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:29.737966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.737890    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:48:29.744040 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:48:29.744005    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fbeb35_7537_43c1_893a_0566e5dc075a.slice/crio-b06a147330bdfabb99c747341a949566281a06b8b9b3544890f6854dfe50761a WatchSource:0}: Error finding container b06a147330bdfabb99c747341a949566281a06b8b9b3544890f6854dfe50761a: Status 404 returned error can't find the container with id b06a147330bdfabb99c747341a949566281a06b8b9b3544890f6854dfe50761a
Nov 03 19:48:29.746057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.746026    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:29.746134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.746096    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:29.746134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.746124    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:29.767284 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:29.767255    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerStarted","Data":"b06a147330bdfabb99c747341a949566281a06b8b9b3544890f6854dfe50761a"}
Nov 03 19:48:30.773711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:30.773673    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerStarted","Data":"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"}
Nov 03 19:48:30.934277 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:30.934228    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podStartSLOduration=2.934215923 podStartE2EDuration="2.934215923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:30.933275007 +0000 UTC m=+3809.210628592" watchObservedRunningTime="2026-11-03 19:48:30.934215923 +0000 UTC m=+3809.211569505"
Nov 03 19:48:31.317464 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:31.317424    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:31.322651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:31.322625    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:31.778798 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:31.778759    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:31.779762 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:31.779745    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:33.891410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:33.891379    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:34.147823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:34.147725    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:35.224015 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:48:35.223976    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:48:37.248543 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.248506    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.498378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.498330    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:37.549226 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.549141    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:48:37.552389 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.552370    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.601579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.601550    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ccf773\""
Nov 03 19:48:37.601753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.601624    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0bf2f12a-dockercfg-r25jf\""
Nov 03 19:48:37.614609 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.614572    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb6gn\" (UniqueName: \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.614780 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.614630    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.647755 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.647716    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:48:37.700990 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.700960    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:48:37.704110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.704082    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.716228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.716204    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb6gn\" (UniqueName: \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.716344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.716236    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.716968 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.716944    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.763130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.763100    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:48:37.775062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.775027    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb6gn\" (UniqueName: \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\") pod \"payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.817067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.816981    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2gq\" (UniqueName: \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.817216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.817135    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.865739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.865700    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:37.918410 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.918359    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.918533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.918469    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2gq\" (UniqueName: \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.919461 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.919394    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:37.976185 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:37.976151    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2gq\" (UniqueName: \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\") pod \"payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:38.016139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.016106    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:38.236351 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.236311    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:48:38.242311 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:48:38.242278    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733bbb81_d49b_4959_bb13_edf6886257bc.slice/crio-717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6 WatchSource:0}: Error finding container 717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6: Status 404 returned error can't find the container with id 717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6
Nov 03 19:48:38.393739 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.393653    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:48:38.400698 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:48:38.400668    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6df27d_46e5_45fa_9f1b_3b3c17ffe325.slice/crio-d9ee4df196ea9bbc1c9aec37202401256b0fb6284645071f6d926e3d9825e6d5 WatchSource:0}: Error finding container d9ee4df196ea9bbc1c9aec37202401256b0fb6284645071f6d926e3d9825e6d5: Status 404 returned error can't find the container with id d9ee4df196ea9bbc1c9aec37202401256b0fb6284645071f6d926e3d9825e6d5
Nov 03 19:48:38.814815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.814759    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" event={"ID":"733bbb81-d49b-4959-bb13-edf6886257bc","Type":"ContainerStarted","Data":"f3078b347a55a11fce69b25ce863015297ae4e805d22331d580053a4469838fa"}
Nov 03 19:48:38.815023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.814873    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" event={"ID":"733bbb81-d49b-4959-bb13-edf6886257bc","Type":"ContainerStarted","Data":"717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6"}
Nov 03 19:48:38.815023 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.814907    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:38.816463 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.816435    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" event={"ID":"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325","Type":"ContainerStarted","Data":"69376013d440fe55fc719aa2037feaad474f813b3a1b8c89fac41ba38c1ad49f"}
Nov 03 19:48:38.816590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.816485    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" event={"ID":"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325","Type":"ContainerStarted","Data":"d9ee4df196ea9bbc1c9aec37202401256b0fb6284645071f6d926e3d9825e6d5"}
Nov 03 19:48:38.816590 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.816515    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:38.926492 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.926443    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" podStartSLOduration=1.926431489 podStartE2EDuration="1.926431489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:38.923900056 +0000 UTC m=+3817.201253638" watchObservedRunningTime="2026-11-03 19:48:38.926431489 +0000 UTC m=+3817.203785071"
Nov 03 19:48:38.930003 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:38.929953    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:39.034180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:39.034126    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" podStartSLOduration=2.0341127 podStartE2EDuration="2.0341127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:48:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:48:39.033280286 +0000 UTC m=+3817.310633862" watchObservedRunningTime="2026-11-03 19:48:39.0341127 +0000 UTC m=+3817.311466283"
Nov 03 19:48:40.388836 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:40.388800    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:40.639860 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:40.639763    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:45.591719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:45.591679    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:49.832833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:49.832769    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:48:49.833371 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:49.832977    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:48:50.557110 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:50.557068    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:55.527729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:55.527689    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:57.910274 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.910247    4944 generic.go:356] "Generic (PLEG): container finished" podID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerID="c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6" exitCode=0
Nov 03 19:48:57.910619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.910315    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerDied","Data":"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"}
Nov 03 19:48:57.910715 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.910701    4944 scope.go:122] "RemoveContainer" containerID="c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"
Nov 03 19:48:57.911906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.911880    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:57.912005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.911946    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:57.912005 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:57.911978    4944 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 03 19:48:58.765407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:58.765373    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:58.934560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:58.934519    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerStarted","Data":"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"}
Nov 03 19:48:59.324802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:59.321971    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:48:59.327938 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:59.327906    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 19:48:59.328172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:59.328146    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 19:48:59.456899 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:59.456864    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:48:59.782144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:48:59.782113    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:00.328266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:00.328226    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 19:49:00.328946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:00.328913    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 19:49:01.028680 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:01.028606    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:01.323871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:01.320306    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 19:49:01.323871 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:01.320399    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 19:49:01.778930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:01.778891    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:49:02.331939 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:02.331890    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 19:49:02.332150 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:02.331952    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 19:49:03.031906 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.031866    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:49:03.035743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.035710    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076400 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076372    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076407    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076440    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076519    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssk7n\" (UniqueName: \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076626    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.076702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.076694    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.083288 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.083259    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:49:03.132170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.132133    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ccf773\"/\"openshift-service-ca.crt\""
Nov 03 19:49:03.132170 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.132154    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ccf773\"/\"kube-root-ca.crt\""
Nov 03 19:49:03.141332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.134028    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ccf773\"/\"simulator-single-main-ccf773-kserve-self-signed-certs\""
Nov 03 19:49:03.141332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.134055    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ccf773\"/\"default-dockercfg-4xsbk\""
Nov 03 19:49:03.177681 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177645    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.177898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177703    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.177898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177726    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.177898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177755    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.177898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177775    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssk7n\" (UniqueName: \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.177898 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.177854    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.178217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.178189    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.178269 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.178237    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.178310 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.178259    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.180050 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.180027    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.180273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.180255    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.204103 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.201377    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:03.241032 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.240997    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssk7n\" (UniqueName: \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\") pod \"simulator-single-main-ccf773-kserve-58c886d86f-bmnqf\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") " pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.326868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.321176    4944 patch_prober.go:29] interesting pod/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 19:49:03.326868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.321240    4944 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 19:49:03.354179 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.354128    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:03.735328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.735246    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:49:03.739753 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:49:03.739720    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96d64d7d_b8e1_4bee_b44a_a402f45c05bd.slice/crio-52194d384b08474d3ea25052bc651a77a92954c58a931a003e230caf3eb9c23e WatchSource:0}: Error finding container 52194d384b08474d3ea25052bc651a77a92954c58a931a003e230caf3eb9c23e: Status 404 returned error can't find the container with id 52194d384b08474d3ea25052bc651a77a92954c58a931a003e230caf3eb9c23e
Nov 03 19:49:03.961172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.961132    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" event={"ID":"96d64d7d-b8e1-4bee-b44a-a402f45c05bd","Type":"ContainerStarted","Data":"473743ac1490dbc210f0a857f4a6da3eab7c51e9ff0931904d508b7ae8f2fe55"}
Nov 03 19:49:03.961344 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:03.961185    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" event={"ID":"96d64d7d-b8e1-4bee-b44a-a402f45c05bd","Type":"ContainerStarted","Data":"52194d384b08474d3ea25052bc651a77a92954c58a931a003e230caf3eb9c23e"}
Nov 03 19:49:04.087002 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:04.086947    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" podStartSLOduration=2.086931902 podStartE2EDuration="2.086931902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:49:04.085849732 +0000 UTC m=+3842.363203317" watchObservedRunningTime="2026-11-03 19:49:04.086931902 +0000 UTC m=+3842.364285510"
Nov 03 19:49:04.321555 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:04.321520    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:49:04.322480 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:04.322457    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:49:04.354779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:04.354721    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:05.224676 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:49:05.224637    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:49:06.009666 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:06.009636    4944 scope.go:122] "RemoveContainer" containerID="7df35c2200b4ca4e4772629072e3b5a89c7f2b276767c01e7c594c95648ea362"
Nov 03 19:49:06.018009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:06.017971    4944 scope.go:122] "RemoveContainer" containerID="5782cd667f105a44c20cf307fca10f2d498e94c57de70e561807f4b7c01cb668"
Nov 03 19:49:06.025719 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:06.025701    4944 scope.go:122] "RemoveContainer" containerID="8673a626980b9293b4bbccbe982fa1467de225f5421233da81127f62918e0d85"
Nov 03 19:49:06.357280 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:06.357166    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:11.571865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:11.571824    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:13.354852 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:13.354814    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:13.367313 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:13.367281    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:14.019937 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:14.019901    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:49:16.879028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:16.878995    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:22.137694 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:22.137655    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.057512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:27.057475    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:27.724426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:27.724389    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:28.001444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:28.001408    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:32.267964 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:32.267922    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:33.087490 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.087454    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"]
Nov 03 19:49:33.091551 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.091529    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.171604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.171571    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.171778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.171613    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9df\" (UniqueName: \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.171869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.171769    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.189355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.189325    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:49:33.189534 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.189517    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:49:33.189600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.189554    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:49:33.189600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.189565    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:49:33.191454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.191436    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"]
Nov 03 19:49:33.272334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.272304    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.272745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.272338    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9df\" (UniqueName: \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.272745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.272462    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.273022 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.273004    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.274883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.274865    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.329834 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.329775    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9df\" (UniqueName: \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\") pod \"maas-api-revoke-sub-469481af-682df\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.405111 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.405044    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:33.775083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:33.775052    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"]
Nov 03 19:49:33.781139 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:49:33.781115    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335d34c9_8094_4ef8_975c_d12966b2e670.slice/crio-56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f WatchSource:0}: Error finding container 56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f: Status 404 returned error can't find the container with id 56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f
Nov 03 19:49:34.109401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:34.109371    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df" event={"ID":"335d34c9-8094-4ef8-975c-d12966b2e670","Type":"ContainerStarted","Data":"56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f"}
Nov 03 19:49:35.115530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:35.115498    4944 generic.go:356] "Generic (PLEG): container finished" podID="335d34c9-8094-4ef8-975c-d12966b2e670" containerID="bb6e732ead6fcf6e6ff8f090fbab3ad81752f2bb792939cb29c600c7e6b04888" exitCode=0
Nov 03 19:49:35.115943 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:35.115561    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df" event={"ID":"335d34c9-8094-4ef8-975c-d12966b2e670","Type":"ContainerDied","Data":"bb6e732ead6fcf6e6ff8f090fbab3ad81752f2bb792939cb29c600c7e6b04888"}
Nov 03 19:49:35.223212 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:49:35.223178    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:49:36.258285 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.258257    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:36.296799 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.296761    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\") pod \"335d34c9-8094-4ef8-975c-d12966b2e670\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") "
Nov 03 19:49:36.296932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.296872    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\" (UniqueName: \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\") pod \"335d34c9-8094-4ef8-975c-d12966b2e670\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") "
Nov 03 19:49:36.296932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.296921    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\") pod \"335d34c9-8094-4ef8-975c-d12966b2e670\" (UID: \"335d34c9-8094-4ef8-975c-d12966b2e670\") "
Nov 03 19:49:36.297304 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.297280    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca" pod "335d34c9-8094-4ef8-975c-d12966b2e670" (UID: "335d34c9-8094-4ef8-975c-d12966b2e670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:36.298831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.298783    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca" pod "335d34c9-8094-4ef8-975c-d12966b2e670" (UID: "335d34c9-8094-4ef8-975c-d12966b2e670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:49:36.298921 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.298848    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df" pod "335d34c9-8094-4ef8-975c-d12966b2e670" (UID: "335d34c9-8094-4ef8-975c-d12966b2e670"). InnerVolumeSpecName "kube-api-access-5d9df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:36.398134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.398054    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:36.398134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.398084    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9df\" (UniqueName: \"kubernetes.io/projected/335d34c9-8094-4ef8-975c-d12966b2e670-kube-api-access-5d9df\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:36.398134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:36.398095    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/335d34c9-8094-4ef8-975c-d12966b2e670-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:37.126416 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:37.126381    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"
Nov 03 19:49:37.126604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:37.126390    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df" event={"ID":"335d34c9-8094-4ef8-975c-d12966b2e670","Type":"ContainerDied","Data":"56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f"}
Nov 03 19:49:37.126604 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:37.126523    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56dc52a27bfae4437f1b70d37d44f04c31be185d20d84742d48ecf92b27b034f"
Nov 03 19:49:37.262648 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:37.262616    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.241821 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:39.241774    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:39.502566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:39.502476    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:42.411302 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:42.411269    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.443191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.443154    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"]
Nov 03 19:49:44.443580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.443572    4944 cpu_manager.go:414] "containerMap: removing container" podUID="335d34c9-8094-4ef8-975c-d12966b2e670" containerName="revoke-keys"
Nov 03 19:49:44.443624 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.443582    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="335d34c9-8094-4ef8-975c-d12966b2e670" containerName="revoke-keys"
Nov 03 19:49:44.443687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.443676    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="335d34c9-8094-4ef8-975c-d12966b2e670" containerName="revoke-keys"
Nov 03 19:49:44.446224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.446201    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.470067 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.470043    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.470160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.470086    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzqw\" (UniqueName: \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.470160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.470154    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.497385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.497366    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:49:44.538447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.538422    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:44.543621 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.543604    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:49:44.543729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.543677    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:49:44.543802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.543746    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:49:44.545760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.545743    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"]
Nov 03 19:49:44.571034 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.571010    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzqw\" (UniqueName: \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.571116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.571051    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.571159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.571135    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.571702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.571682    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.573583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.573560    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.628548 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.628512    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzqw\" (UniqueName: \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:44.757842 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:44.757814    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:45.125633 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:45.125538    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"]
Nov 03 19:49:45.132610 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:49:45.132581    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b5f78a_b757_4fc5_afc4_6d0f4b0a0a66.slice/crio-1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281 WatchSource:0}: Error finding container 1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281: Status 404 returned error can't find the container with id 1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281
Nov 03 19:49:45.165484 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:45.165445    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc" event={"ID":"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66","Type":"ContainerStarted","Data":"1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281"}
Nov 03 19:49:46.171736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:46.171702    4944 generic.go:356] "Generic (PLEG): container finished" podID="36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" containerID="bf2433b410e5c29c5cdb6e21c4430faa2f6794fb5e18e0506a55016ac1be55a4" exitCode=0
Nov 03 19:49:46.172187 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:46.171770    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc" event={"ID":"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66","Type":"ContainerDied","Data":"bf2433b410e5c29c5cdb6e21c4430faa2f6794fb5e18e0506a55016ac1be55a4"}
Nov 03 19:49:47.308364 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.308339    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:47.395425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.395389    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\") pod \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") "
Nov 03 19:49:47.395615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.395452    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\" (UniqueName: \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\") pod \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") "
Nov 03 19:49:47.395615 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.395542    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\") pod \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\" (UID: \"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66\") "
Nov 03 19:49:47.395855 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.395832    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca" pod "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" (UID: "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:49:47.395956 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.395940    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.397567 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.397542    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca" pod "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" (UID: "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:49:47.397697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.397679    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw" pod "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" (UID: "36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66"). InnerVolumeSpecName "kube-api-access-snzqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:49:47.496606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.496571    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snzqw\" (UniqueName: \"kubernetes.io/projected/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-kube-api-access-snzqw\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.496606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.496603    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:49:47.780940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:47.780850    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:48.183486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:48.183406    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"
Nov 03 19:49:48.183486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:48.183453    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc" event={"ID":"36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66","Type":"ContainerDied","Data":"1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281"}
Nov 03 19:49:48.183687 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:48.183509    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cb3a4c32552705c1da3b3fac292c29ae6d6b7426db40ee73c507e083f536281"
Nov 03 19:49:49.123162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:49.123130    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:49:49.325035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:49.325004    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:52.807894 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:52.807859    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:56.057805 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:56.057755    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:49:56.158595 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:56.158561    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:49:56.509909 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:56.509868    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:56.757593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:56.757563    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:49:57.821930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:49:57.821892    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:01.159476 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:01.159428    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing" containerID="cri-o://f3078b347a55a11fce69b25ce863015297ae4e805d22331d580053a4469838fa" gracePeriod=25
Nov 03 19:50:01.223128 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:01.223086    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing" containerID="cri-o://69376013d440fe55fc719aa2037feaad474f813b3a1b8c89fac41ba38c1ad49f" gracePeriod=25
Nov 03 19:50:02.186383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:02.186351    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:02.542076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:02.542047    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:50:02.542380 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:02.542364    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:50:03.789574 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:03.789533    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.076853 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.076753    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:04.143026 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.142984    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" podUID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" containerName="main" containerID="cri-o://473743ac1490dbc210f0a857f4a6da3eab7c51e9ff0931904d508b7ae8f2fe55" gracePeriod=45
Nov 03 19:50:04.268299 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.268266    4944 generic.go:356] "Generic (PLEG): container finished" podID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" containerID="473743ac1490dbc210f0a857f4a6da3eab7c51e9ff0931904d508b7ae8f2fe55" exitCode=0
Nov 03 19:50:04.268458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.268333    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" event={"ID":"96d64d7d-b8e1-4bee-b44a-a402f45c05bd","Type":"ContainerDied","Data":"473743ac1490dbc210f0a857f4a6da3eab7c51e9ff0931904d508b7ae8f2fe55"}
Nov 03 19:50:04.322134 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.322100    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"]
Nov 03 19:50:04.366074 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.365997    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ccf773-6z9fc"]
Nov 03 19:50:04.393749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.393724    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:50:04.460957 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.460928    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.460985    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461014    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461063    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\" (UniqueName: \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461093    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461110    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\") pod \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\" (UID: \"96d64d7d-b8e1-4bee-b44a-a402f45c05bd\") "
Nov 03 19:50:04.461430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461249    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:04.461430 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461331    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:04.461517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461433    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:04.461601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461576    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.461639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461617    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.461639 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.461633    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.463216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.463193    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:04.463319 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.463278    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:50:04.463398 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.463369    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n" pod "96d64d7d-b8e1-4bee-b44a-a402f45c05bd" (UID: "96d64d7d-b8e1-4bee-b44a-a402f45c05bd"). InnerVolumeSpecName "kube-api-access-ssk7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:04.563059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.563030    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssk7n\" (UniqueName: \"kubernetes.io/projected/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-kube-api-access-ssk7n\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.563059 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.563057    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.563262 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.563067    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96d64d7d-b8e1-4bee-b44a-a402f45c05bd-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:04.843895 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:04.843863    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:05.223388 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:50:05.223302    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:50:05.274421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.274385    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf" event={"ID":"96d64d7d-b8e1-4bee-b44a-a402f45c05bd","Type":"ContainerDied","Data":"52194d384b08474d3ea25052bc651a77a92954c58a931a003e230caf3eb9c23e"}
Nov 03 19:50:05.274421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.274411    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"
Nov 03 19:50:05.274622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.274446    4944 scope.go:122] "RemoveContainer" containerID="473743ac1490dbc210f0a857f4a6da3eab7c51e9ff0931904d508b7ae8f2fe55"
Nov 03 19:50:05.428778 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.428747    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:50:05.475071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.475004    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ccf773/simulator-single-main-ccf773-kserve-58c886d86f-bmnqf"]
Nov 03 19:50:05.933729 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.933695    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" path="/var/lib/kubelet/pods/36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66/volumes"
Nov 03 19:50:05.934126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:05.934104    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" path="/var/lib/kubelet/pods/96d64d7d-b8e1-4bee-b44a-a402f45c05bd/volumes"
Nov 03 19:50:06.053863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:06.053833    4944 scope.go:122] "RemoveContainer" containerID="c1f960d5a040a15ed0f2010d46b6949c619ec00e4a80664f417c824f3da8d83c"
Nov 03 19:50:06.062221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:06.062196    4944 scope.go:122] "RemoveContainer" containerID="c395b0ca5482b3487e87d4432c7aa68333c9809f7c70e035867b0814e0bf164e"
Nov 03 19:50:06.070359 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:06.070339    4944 scope.go:122] "RemoveContainer" containerID="1d4d62ca8a13a18f29a71c7bc7a81a26db86320fc49c2f3ac8a920a16f5b9473"
Nov 03 19:50:09.823811 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:09.823747    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:50:09.824209 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:09.823761    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:50:10.786944 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:10.786910    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:10.991316 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:10.991246    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:50:10.991671 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:10.991456    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy" containerID="cri-o://a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9" gracePeriod=30
Nov 03 19:50:11.038323 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:11.038241    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:11.293483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:11.293397    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:16.236141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.236118    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:50:16.326355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.326282    4944 generic.go:356] "Generic (PLEG): container finished" podID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerID="a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9" exitCode=0
Nov 03 19:50:16.326355 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.326326    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"
Nov 03 19:50:16.326539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.326321    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerDied","Data":"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"}
Nov 03 19:50:16.326539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.326442    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7" event={"ID":"22fbeb35-7537-43c1-893a-0566e5dc075a","Type":"ContainerDied","Data":"b06a147330bdfabb99c747341a949566281a06b8b9b3544890f6854dfe50761a"}
Nov 03 19:50:16.326539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.326473    4944 scope.go:122] "RemoveContainer" containerID="a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"
Nov 03 19:50:16.334919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.334858    4944 scope.go:122] "RemoveContainer" containerID="c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"
Nov 03 19:50:16.342444 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.342427    4944 scope.go:122] "RemoveContainer" containerID="a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"
Nov 03 19:50:16.342727 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:50:16.342710    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9\": container with ID starting with a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9 not found: ID does not exist" containerID="a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"
Nov 03 19:50:16.342806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.342743    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9"} err="failed to get container status \"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9\": rpc error: code = NotFound desc = could not find container \"a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9\": container with ID starting with a2bfb31b36d280f0a477b4e82fa5ade3091812fdfb7abe631b7b0f06d7cebbc9 not found: ID does not exist"
Nov 03 19:50:16.342806 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.342760    4944 scope.go:122] "RemoveContainer" containerID="c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"
Nov 03 19:50:16.343068 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:50:16.343050    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6\": container with ID starting with c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6 not found: ID does not exist" containerID="c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"
Nov 03 19:50:16.343119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.343074    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6"} err="failed to get container status \"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6\": rpc error: code = NotFound desc = could not find container \"c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6\": container with ID starting with c5236f5fa38eaeac69df4358c63f4a84bb125331b5c9616264aadc9dbbbd2ed6 not found: ID does not exist"
Nov 03 19:50:16.377402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377380    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377422    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377447    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377468    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377530    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377557    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377616 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377601    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377654    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377710    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:16.377779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377719    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\") pod \"22fbeb35-7537-43c1-893a-0566e5dc075a\" (UID: \"22fbeb35-7537-43c1-893a-0566e5dc075a\") "
Nov 03 19:50:16.377995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377928    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:16.377995 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.377984    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:16.378121 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378071    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:16.378180 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378128    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:16.378320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378293    4944 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-credential-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.378320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378320    4944 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-socket\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.378566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378335    4944 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-data\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.378566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378350    4944 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-workload-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.378566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.378364    4944 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fbeb35-7537-43c1-893a-0566e5dc075a-istiod-ca-cert\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.379652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.379624    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "kube-api-access-rkq64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:16.379768 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.379749    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 19:50:16.379887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.379849    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:50:16.379887 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.379854    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token" pod "22fbeb35-7537-43c1-893a-0566e5dc075a" (UID: "22fbeb35-7537-43c1-893a-0566e5dc075a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:16.479265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.479232    4944 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-token\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.479265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.479260    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq64\" (UniqueName: \"kubernetes.io/projected/22fbeb35-7537-43c1-893a-0566e5dc075a-kube-api-access-rkq64\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.479265 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.479271    4944 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-podinfo\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.479500 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.479279    4944 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fbeb35-7537-43c1-893a-0566e5dc075a-istio-envoy\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:16.740514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.740476    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:50:16.744374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.744341    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ccf773-openshift-default-655768b49-vlpp7"]
Nov 03 19:50:16.855317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855281    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:50:16.855744 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855725    4944 cpu_manager.go:414] "containerMap: removing container" podUID="36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" containerName="revoke-keys"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855746    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" containerName="revoke-keys"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855763    4944 cpu_manager.go:414] "containerMap: removing container" podUID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" containerName="main"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855771    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" containerName="main"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855801    4944 cpu_manager.go:414] "containerMap: removing container" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855809    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855847    4944 cpu_manager.go:414] "containerMap: removing container" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.855882 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855854    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.856255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855975    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.856255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.855993    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b5f78a-b757-4fc5-afc4-6d0f4b0a0a66" containerName="revoke-keys"
Nov 03 19:50:16.856255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.856004    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="96d64d7d-b8e1-4bee-b44a-a402f45c05bd" containerName="main"
Nov 03 19:50:16.856255 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.856016    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" containerName="istio-proxy"
Nov 03 19:50:16.860466 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.860444    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882306    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882337    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882448 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882370    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swm6v\" (UniqueName: \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882539 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882453    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882548    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.882586 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.882570    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.909701 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.909631    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8840718e-kserve-self-signed-certs\""
Nov 03 19:50:16.919117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.919089    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:50:16.983618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983580    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.983618 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983619    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.983915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983673    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.983915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983695    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.983915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983747    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swm6v\" (UniqueName: \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.983915 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.983776    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.984160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.984046    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.984160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.984067    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.984160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.984144    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.985985 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.985963    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:16.986267 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:16.986248    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:17.041230 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:17.041194    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swm6v\" (UniqueName: \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\") pod \"e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") " pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:17.057632 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:17.057602    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:17.172407 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:17.172325    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:17.539872 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:17.539836    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:50:17.547366 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:50:17.547323    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6243b563_e2c4_4d4a_a5ed_3cfa8e3126b6.slice/crio-f9c16bfdaaf8c237fb07758b4ebd1a41efeb02503fd3800d54f824d4d20e2b5a WatchSource:0}: Error finding container f9c16bfdaaf8c237fb07758b4ebd1a41efeb02503fd3800d54f824d4d20e2b5a: Status 404 returned error can't find the container with id f9c16bfdaaf8c237fb07758b4ebd1a41efeb02503fd3800d54f824d4d20e2b5a
Nov 03 19:50:17.933396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:17.933320    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fbeb35-7537-43c1-893a-0566e5dc075a" path="/var/lib/kubelet/pods/22fbeb35-7537-43c1-893a-0566e5dc075a/volumes"
Nov 03 19:50:18.340411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:18.340378    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" event={"ID":"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6","Type":"ContainerStarted","Data":"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"}
Nov 03 19:50:18.340411 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:18.340418    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" event={"ID":"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6","Type":"ContainerStarted","Data":"f9c16bfdaaf8c237fb07758b4ebd1a41efeb02503fd3800d54f824d4d20e2b5a"}
Nov 03 19:50:18.450368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:18.450317    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" podStartSLOduration=2.45030506 podStartE2EDuration="2.45030506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:50:18.449402185 +0000 UTC m=+3916.726755771" watchObservedRunningTime="2026-11-03 19:50:18.45030506 +0000 UTC m=+3916.727658643"
Nov 03 19:50:19.172713 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:19.172684    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:19.824334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:19.824290    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:50:19.824334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:19.824318    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 19:50:21.358324 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.358277    4944 generic.go:356] "Generic (PLEG): container finished" podID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerID="69376013d440fe55fc719aa2037feaad474f813b3a1b8c89fac41ba38c1ad49f" exitCode=0
Nov 03 19:50:21.358835 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.358411    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" event={"ID":"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325","Type":"ContainerDied","Data":"69376013d440fe55fc719aa2037feaad474f813b3a1b8c89fac41ba38c1ad49f"}
Nov 03 19:50:21.360320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.360295    4944 generic.go:356] "Generic (PLEG): container finished" podID="733bbb81-d49b-4959-bb13-edf6886257bc" containerID="f3078b347a55a11fce69b25ce863015297ae4e805d22331d580053a4469838fa" exitCode=0
Nov 03 19:50:21.360447 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.360330    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" event={"ID":"733bbb81-d49b-4959-bb13-edf6886257bc","Type":"ContainerDied","Data":"f3078b347a55a11fce69b25ce863015297ae4e805d22331d580053a4469838fa"}
Nov 03 19:50:21.463699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.463674    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:50:21.499507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.499485    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:50:21.522815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.522783    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\" (UniqueName: \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\") pod \"733bbb81-d49b-4959-bb13-edf6886257bc\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") "
Nov 03 19:50:21.522934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.522821    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\") pod \"733bbb81-d49b-4959-bb13-edf6886257bc\" (UID: \"733bbb81-d49b-4959-bb13-edf6886257bc\") "
Nov 03 19:50:21.522934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.522877    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\" (UniqueName: \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\") pod \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") "
Nov 03 19:50:21.522934 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.522918    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\") pod \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\" (UID: \"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325\") "
Nov 03 19:50:21.523261 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.523233    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume" pod "733bbb81-d49b-4959-bb13-edf6886257bc" (UID: "733bbb81-d49b-4959-bb13-edf6886257bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:21.523375 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.523271    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume" pod "0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" (UID: "0d6df27d-46e5-45fa-9f1b-3b3c17ffe325"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:50:21.524986 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.524965    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn" pod "733bbb81-d49b-4959-bb13-edf6886257bc" (UID: "733bbb81-d49b-4959-bb13-edf6886257bc"). InnerVolumeSpecName "kube-api-access-sb6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:21.525035 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.524997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq" pod "0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" (UID: "0d6df27d-46e5-45fa-9f1b-3b3c17ffe325"). InnerVolumeSpecName "kube-api-access-8x2gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:50:21.623843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.623815    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb6gn\" (UniqueName: \"kubernetes.io/projected/733bbb81-d49b-4959-bb13-edf6886257bc-kube-api-access-sb6gn\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:21.623843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.623839    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733bbb81-d49b-4959-bb13-edf6886257bc-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:21.623843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.623848    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x2gq\" (UniqueName: \"kubernetes.io/projected/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-kube-api-access-8x2gq\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:21.624081 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:21.623857    4944 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325-plugins-config-volume\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:50:21.973215 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:50:21.973170    4944 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733bbb81_d49b_4959_bb13_edf6886257bc.slice/crio-717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6df27d_46e5_45fa_9f1b_3b3c17ffe325.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733bbb81_d49b_4959_bb13_edf6886257bc.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 19:50:22.367233 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.367206    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"
Nov 03 19:50:22.367728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.367204    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286" event={"ID":"733bbb81-d49b-4959-bb13-edf6886257bc","Type":"ContainerDied","Data":"717ab39763695793f9d26b10a39ba4a5688f4c94bcc0b1a6d40f8aa27e448da6"}
Nov 03 19:50:22.367728 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.367357    4944 scope.go:122] "RemoveContainer" containerID="f3078b347a55a11fce69b25ce863015297ae4e805d22331d580053a4469838fa"
Nov 03 19:50:22.368830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.368705    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"
Nov 03 19:50:22.368830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.368742    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb" event={"ID":"0d6df27d-46e5-45fa-9f1b-3b3c17ffe325","Type":"ContainerDied","Data":"d9ee4df196ea9bbc1c9aec37202401256b0fb6284645071f6d926e3d9825e6d5"}
Nov 03 19:50:22.378643 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.378619    4944 scope.go:122] "RemoveContainer" containerID="69376013d440fe55fc719aa2037feaad474f813b3a1b8c89fac41ba38c1ad49f"
Nov 03 19:50:22.522965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.522901    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:50:22.526523 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.526490    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ccf773-674698f98xfmlb"]
Nov 03 19:50:22.634088 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.634001    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:50:22.639771 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:22.639740    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ccf773-6ddb7z6286"]
Nov 03 19:50:23.958815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:23.954509    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" path="/var/lib/kubelet/pods/0d6df27d-46e5-45fa-9f1b-3b3c17ffe325/volumes"
Nov 03 19:50:23.958815 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:23.956309    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" path="/var/lib/kubelet/pods/733bbb81-d49b-4959-bb13-edf6886257bc/volumes"
Nov 03 19:50:24.739978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:24.739936    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-gwxrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:24.740239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:24.740207    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-gwxrp" podUID="55c38327-6415-43dc-b53b-8d2033f1bd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:24.860818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:24.859938    4944 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-stw9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 19:50:24.860818 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:24.860041    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-stw9w" podUID="19b4623f-6a4c-4ed0-bad9-7883737d362a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.12:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 19:50:25.050869 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:25.050176    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"]
Nov 03 19:50:25.095384 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:25.095340    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e8a3dd-rmmn6"]
Nov 03 19:50:25.818571 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:25.818528    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:25.933205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:25.933168    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7" path="/var/lib/kubelet/pods/ee3af2ac-6c2f-4dcd-84ff-7324cfae77d7/volumes"
Nov 03 19:50:26.698414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:26.698379    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:27.173292 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:27.173257    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:27.185745 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:27.185712    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:27.410437 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:27.410406    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:50:35.224488 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:50:35.224450    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:50:39.651752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:39.651709    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:41.498922 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:41.498877    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:54.961454 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:54.961411    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:58.858748 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.858711    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:50:58.859205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859165    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing"
Nov 03 19:50:58.859205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859178    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing"
Nov 03 19:50:58.859205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859187    4944 cpu_manager.go:414] "containerMap: removing container" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing"
Nov 03 19:50:58.859205 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859193    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing"
Nov 03 19:50:58.859362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859253    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d6df27d-46e5-45fa-9f1b-3b3c17ffe325" containerName="payload-processing"
Nov 03 19:50:58.859362 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.859261    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="733bbb81-d49b-4959-bb13-edf6886257bc" containerName="payload-pre-processing"
Nov 03 19:50:58.862222 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.862199    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:58.957620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.957584    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8840718e-kserve-self-signed-certs\""
Nov 03 19:50:58.959620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:58.959598    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:50:59.012174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012134    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r595q\" (UniqueName: \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.012385 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012201    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.012455 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012383    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.012515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012500    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.012645 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012622    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.012731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.012660    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.059054 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.059018    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:50:59.113845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113749    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r595q\" (UniqueName: \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.113845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113807    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113859    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113881    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113907    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.113921    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.114207    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114279 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.114223    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.114462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.114440    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.116240 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.116215    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.116507 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.116488    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.171920 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.171890    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r595q\" (UniqueName: \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\") pod \"e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") " pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.173761 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.173734    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:50:59.547512 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:50:59.547473    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:50:59.554678 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:50:59.554649    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9010198_cb29_4513_a776_db4a17c8c0ed.slice/crio-ce9300a8aa1dd0b8ac21fa04a7ca8b722215c206c0ac8dd939af7db1c1590f6c WatchSource:0}: Error finding container ce9300a8aa1dd0b8ac21fa04a7ca8b722215c206c0ac8dd939af7db1c1590f6c: Status 404 returned error can't find the container with id ce9300a8aa1dd0b8ac21fa04a7ca8b722215c206c0ac8dd939af7db1c1590f6c
Nov 03 19:51:00.557993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:00.557954    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" event={"ID":"a9010198-cb29-4513-a776-db4a17c8c0ed","Type":"ContainerStarted","Data":"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"}
Nov 03 19:51:00.557993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:00.557998    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" event={"ID":"a9010198-cb29-4513-a776-db4a17c8c0ed","Type":"ContainerStarted","Data":"ce9300a8aa1dd0b8ac21fa04a7ca8b722215c206c0ac8dd939af7db1c1590f6c"}
Nov 03 19:51:00.669594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:00.669538    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" podStartSLOduration=2.669523959 podStartE2EDuration="2.669523959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:00.667475359 +0000 UTC m=+3958.944828942" watchObservedRunningTime="2026-11-03 19:51:00.669523959 +0000 UTC m=+3958.946877541"
Nov 03 19:51:01.174606 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:01.174570    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:51:05.231587 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:51:05.231544    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:51:08.044998 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:08.044955    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"]
Nov 03 19:51:08.091378 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:08.091331    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rbv46"]
Nov 03 19:51:09.174368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:09.174331    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:51:09.186873 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:09.186846    4944 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:51:09.622576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:09.622543    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:51:09.932117 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:09.932040    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5fd0b7-557c-4703-8886-f43206fe39cd" path="/var/lib/kubelet/pods/3f5fd0b7-557c-4703-8886-f43206fe39cd/volumes"
Nov 03 19:51:16.087930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:16.087867    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:16.333962 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:16.333932    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:29.234161 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.234121    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"]
Nov 03 19:51:29.239104 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.239077    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.288076 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.288047    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:51:29.302576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.302545    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.302731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.302705    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmh69\" (UniqueName: \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.302809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.302755    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.334478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.334441    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:51:29.334612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.334476    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:51:29.334612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.334496    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:51:29.336369 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.336347    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"]
Nov 03 19:51:29.403710 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.403680    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.403859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.403761    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmh69\" (UniqueName: \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.403859 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.403823    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.404374 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.404352    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.406291 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.406270    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.461325 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.461297    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmh69\" (UniqueName: \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\") pod \"maas-api-revoke-sub-452b6d7b-zv87c\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.551765 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.551737    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:29.916483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:29.916398    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"]
Nov 03 19:51:29.924079 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:51:29.924050    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3538d60_e0e8_4a71_9ced_0197bc7e45a7.slice/crio-3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f WatchSource:0}: Error finding container 3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f: Status 404 returned error can't find the container with id 3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f
Nov 03 19:51:30.711980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:30.711944    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c" event={"ID":"b3538d60-e0e8-4a71-9ced-0197bc7e45a7","Type":"ContainerStarted","Data":"3092608ebadf1089b44ce3a7977e7151a5a5a94cb2042484ef3c675c5d297c30"}
Nov 03 19:51:30.712404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:30.711997    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c" event={"ID":"b3538d60-e0e8-4a71-9ced-0197bc7e45a7","Type":"ContainerStarted","Data":"3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f"}
Nov 03 19:51:30.820948 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:30.820904    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c" podStartSLOduration=1.8208916720000001 podStartE2EDuration="1.820891672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:51:30.820708731 +0000 UTC m=+3989.098062309" watchObservedRunningTime="2026-11-03 19:51:30.820891672 +0000 UTC m=+3989.098245254"
Nov 03 19:51:31.720241 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:31.720194    4944 generic.go:356] "Generic (PLEG): container finished" podID="b3538d60-e0e8-4a71-9ced-0197bc7e45a7" containerID="3092608ebadf1089b44ce3a7977e7151a5a5a94cb2042484ef3c675c5d297c30" exitCode=0
Nov 03 19:51:31.720824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:31.720273    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c" event={"ID":"b3538d60-e0e8-4a71-9ced-0197bc7e45a7","Type":"ContainerDied","Data":"3092608ebadf1089b44ce3a7977e7151a5a5a94cb2042484ef3c675c5d297c30"}
Nov 03 19:51:33.104576 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.104552    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:33.189764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.189720    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\") pod \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") "
Nov 03 19:51:33.189974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.189858    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\" (UniqueName: \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\") pod \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") "
Nov 03 19:51:33.189974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.189881    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\") pod \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\" (UID: \"b3538d60-e0e8-4a71-9ced-0197bc7e45a7\") "
Nov 03 19:51:33.190321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.190292    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca" pod "b3538d60-e0e8-4a71-9ced-0197bc7e45a7" (UID: "b3538d60-e0e8-4a71-9ced-0197bc7e45a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:51:33.192462 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.192440    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca" pod "b3538d60-e0e8-4a71-9ced-0197bc7e45a7" (UID: "b3538d60-e0e8-4a71-9ced-0197bc7e45a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:33.192531 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.192483    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69" pod "b3538d60-e0e8-4a71-9ced-0197bc7e45a7" (UID: "b3538d60-e0e8-4a71-9ced-0197bc7e45a7"). InnerVolumeSpecName "kube-api-access-fmh69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:33.290925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.290888    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:33.290925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.290920    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmh69\" (UniqueName: \"kubernetes.io/projected/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-kube-api-access-fmh69\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:33.290925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.290932    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3538d60-e0e8-4a71-9ced-0197bc7e45a7-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:33.777674 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.777634    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c" event={"ID":"b3538d60-e0e8-4a71-9ced-0197bc7e45a7","Type":"ContainerDied","Data":"3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f"}
Nov 03 19:51:33.777980 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.777957    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a2b4c195913e6b852db3f0d1f145099c7c5c5db8a88df9e0d35a0ca0d3e552f"
Nov 03 19:51:33.778174 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:33.778156    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"
Nov 03 19:51:35.224467 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:51:35.224436    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:51:39.972747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:39.972710    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:42.290169 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:42.290082    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:51:42.428649 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:42.428619    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.309866 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.309779    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerName="main" containerID="cri-o://ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2" gracePeriod=45
Nov 03 19:51:57.400044 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.400005    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerName="main" probeResult="failure" output="Get \"https://10.134.1.43:8000/ready\": dial tcp 10.134.1.43:8000: connect: connection refused"
Nov 03 19:51:57.484305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.484271    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:51:57.564273 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.564210    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:51:57.608030 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.607994    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:51:57.762052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762016    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762058    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762120    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762140    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762225 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762163    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\" (UniqueName: \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762237    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\") pod \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\" (UID: \"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6\") "
Nov 03 19:51:57.762473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762291    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.762473 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762398    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.762652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762557    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.762652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762580    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.762652 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.762598    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.764396 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.764376    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:51:57.764514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.764431    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:51:57.764556 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.764523    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v" pod "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" (UID: "6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6"). InnerVolumeSpecName "kube-api-access-swm6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:51:57.863515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.863425    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.863515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.863470    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swm6v\" (UniqueName: \"kubernetes.io/projected/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-kube-api-access-swm6v\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.863515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.863502    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.863515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.863512    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:51:57.909073 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.909039    4944 generic.go:356] "Generic (PLEG): container finished" podID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerID="ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2" exitCode=0
Nov 03 19:51:57.909228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.909086    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"
Nov 03 19:51:57.909228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.909117    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" event={"ID":"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6","Type":"ContainerDied","Data":"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"}
Nov 03 19:51:57.909228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.909160    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf" event={"ID":"6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6","Type":"ContainerDied","Data":"f9c16bfdaaf8c237fb07758b4ebd1a41efeb02503fd3800d54f824d4d20e2b5a"}
Nov 03 19:51:57.909228 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.909178    4944 scope.go:122] "RemoveContainer" containerID="ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"
Nov 03 19:51:57.918353 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.918336    4944 scope.go:122] "RemoveContainer" containerID="ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"
Nov 03 19:51:57.918595 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:51:57.918578    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2\": container with ID starting with ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2 not found: ID does not exist" containerID="ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"
Nov 03 19:51:57.918647 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:57.918603    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2"} err="failed to get container status \"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2\": rpc error: code = NotFound desc = could not find container \"ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2\": container with ID starting with ddb953c6c0c99fbcae8adcbe4fc10c936638de6ef8a2dacbd469012a5b8efba2 not found: ID does not exist"
Nov 03 19:51:58.061900 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:58.061863    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:51:58.067628 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:58.067600    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8840718e-kserve-6ccb858497-zj9bf"]
Nov 03 19:51:59.932901 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:51:59.932867    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" path="/var/lib/kubelet/pods/6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6/volumes"
Nov 03 19:52:01.659698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.659662    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"]
Nov 03 19:52:01.660217 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660201    4944 cpu_manager.go:414] "containerMap: removing container" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerName="main"
Nov 03 19:52:01.660266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660219    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerName="main"
Nov 03 19:52:01.660266 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660261    4944 cpu_manager.go:414] "containerMap: removing container" podUID="b3538d60-e0e8-4a71-9ced-0197bc7e45a7" containerName="revoke-keys"
Nov 03 19:52:01.660333 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660268    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3538d60-e0e8-4a71-9ced-0197bc7e45a7" containerName="revoke-keys"
Nov 03 19:52:01.660368 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660362    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="6243b563-e2c4-4d4a-a5ed-3cfa8e3126b6" containerName="main"
Nov 03 19:52:01.660402 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.660379    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3538d60-e0e8-4a71-9ced-0197bc7e45a7" containerName="revoke-keys"
Nov 03 19:52:01.665182 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.665158    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.690903 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.690875    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lbx5\" (UniqueName: \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.691060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.690917    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.691060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.690944    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.762141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.762071    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:52:01.762141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.762115    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:52:01.762141 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.762126    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:52:01.762391 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.762178    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:52:01.764084 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.764063    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"]
Nov 03 19:52:01.791579 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.791551    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.791722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.791638    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lbx5\" (UniqueName: \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.791722 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.791671    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.792238 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.792213    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.794014 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.793987    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:01.849421 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:01.849389    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lbx5\" (UniqueName: \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\") pod \"maas-api-revoke-sub-3e31a506-qlnk9\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:02.031125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:02.031091    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:52:02.041868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:02.041845    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:02.412740 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:02.412636    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"]
Nov 03 19:52:02.420727 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:52:02.420671    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9cd939_4d90_4281_9868_65004ae59e32.slice/crio-74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc WatchSource:0}: Error finding container 74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc: Status 404 returned error can't find the container with id 74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc
Nov 03 19:52:02.940191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:02.940145    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9" event={"ID":"0f9cd939-4d90-4281-9868-65004ae59e32","Type":"ContainerStarted","Data":"ab3be95ac0fa3fde6cf0a27fe3af39e3166801d52caa3b5bac38fbfbf1aabdb8"}
Nov 03 19:52:02.940717 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:02.940215    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9" event={"ID":"0f9cd939-4d90-4281-9868-65004ae59e32","Type":"ContainerStarted","Data":"74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc"}
Nov 03 19:52:03.050078 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:03.050017    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9" podStartSLOduration=2.050004163 podStartE2EDuration="2.050004163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:52:03.04855106 +0000 UTC m=+4021.325904643" watchObservedRunningTime="2026-11-03 19:52:03.050004163 +0000 UTC m=+4021.327357746"
Nov 03 19:52:03.978057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:03.978010    4944 generic.go:356] "Generic (PLEG): container finished" podID="0f9cd939-4d90-4281-9868-65004ae59e32" containerID="ab3be95ac0fa3fde6cf0a27fe3af39e3166801d52caa3b5bac38fbfbf1aabdb8" exitCode=0
Nov 03 19:52:03.978580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:03.978080    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9" event={"ID":"0f9cd939-4d90-4281-9868-65004ae59e32","Type":"ContainerDied","Data":"ab3be95ac0fa3fde6cf0a27fe3af39e3166801d52caa3b5bac38fbfbf1aabdb8"}
Nov 03 19:52:05.117404 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.117379    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:05.181601 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.181571    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\") pod \"0f9cd939-4d90-4281-9868-65004ae59e32\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") "
Nov 03 19:52:05.181736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.181646    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\") pod \"0f9cd939-4d90-4281-9868-65004ae59e32\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") "
Nov 03 19:52:05.181783 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.181742    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\" (UniqueName: \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\") pod \"0f9cd939-4d90-4281-9868-65004ae59e32\" (UID: \"0f9cd939-4d90-4281-9868-65004ae59e32\") "
Nov 03 19:52:05.181981 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.181960    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca" pod "0f9cd939-4d90-4281-9868-65004ae59e32" (UID: "0f9cd939-4d90-4281-9868-65004ae59e32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:05.182115 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.182101    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:05.183683 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.183648    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca" pod "0f9cd939-4d90-4281-9868-65004ae59e32" (UID: "0f9cd939-4d90-4281-9868-65004ae59e32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:05.183800 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.183774    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5" pod "0f9cd939-4d90-4281-9868-65004ae59e32" (UID: "0f9cd939-4d90-4281-9868-65004ae59e32"). InnerVolumeSpecName "kube-api-access-6lbx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:05.222956 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:52:05.222923    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:52:05.283090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.283059    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9cd939-4d90-4281-9868-65004ae59e32-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:05.283090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.283086    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lbx5\" (UniqueName: \"kubernetes.io/projected/0f9cd939-4d90-4281-9868-65004ae59e32-kube-api-access-6lbx5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:05.988904 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.988872    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9"
Nov 03 19:52:05.989085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.988901    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-qlnk9" event={"ID":"0f9cd939-4d90-4281-9868-65004ae59e32","Type":"ContainerDied","Data":"74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc"}
Nov 03 19:52:05.989085 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:05.988948    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74853ce7d24e4238ce788ec9aa75b8f6ea1cd8703a8ab70be7ad9ca4a0e2c3fc"
Nov 03 19:52:06.164438 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:06.164408    4944 scope.go:122] "RemoveContainer" containerID="26c7a8a7bd8250a34ec3732b8206d9eb2ecbe086199d71228dad3801604fde02"
Nov 03 19:52:06.173930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:06.173906    4944 scope.go:122] "RemoveContainer" containerID="e011b60b66df22c167071cd3e603a61bd9ee28abcf05931d51b060aba2cb1e4a"
Nov 03 19:52:06.181393 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:06.181375    4944 scope.go:122] "RemoveContainer" containerID="49c999f2b4d4d048735350bc904d1388ae9a4ef2adf5b24c0fbb5c05de40608e"
Nov 03 19:52:09.043130 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:09.043095    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:12.505497 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.505441    4944 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerName="main" containerID="cri-o://98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2" gracePeriod=45
Nov 03 19:52:12.611116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.611072    4944 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerName="main" probeResult="failure" output="Get \"https://10.134.1.44:8000/ready\": dial tcp 10.134.1.44:8000: connect: connection refused"
Nov 03 19:52:12.751042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.751018    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:52:12.852711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852623    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.852711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852661    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\" (UniqueName: \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.852711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852681    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.853001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852725    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.853001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852762    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.853001 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852905    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\") pod \"a9010198-cb29-4513-a776-db4a17c8c0ed\" (UID: \"a9010198-cb29-4513-a776-db4a17c8c0ed\") "
Nov 03 19:52:12.853189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.852970    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:12.853189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.853016    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:12.853296 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.853202    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:12.853360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.853313    4944 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-model-cache\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:12.853360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.853332    4944 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-home\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:12.853360 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.853349    4944 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-tmp-dir\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:12.854908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.854883    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 19:52:12.855041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.854980    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:12.855041 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.855000    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q" pod "a9010198-cb29-4513-a776-db4a17c8c0ed" (UID: "a9010198-cb29-4513-a776-db4a17c8c0ed"). InnerVolumeSpecName "kube-api-access-r595q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:12.953954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.953920    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r595q\" (UniqueName: \"kubernetes.io/projected/a9010198-cb29-4513-a776-db4a17c8c0ed-kube-api-access-r595q\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:12.953954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.953948    4944 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a9010198-cb29-4513-a776-db4a17c8c0ed-dshm\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:12.953954 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:12.953958    4944 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a9010198-cb29-4513-a776-db4a17c8c0ed-tls-certs\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:13.023532 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.023491    4944 generic.go:356] "Generic (PLEG): container finished" podID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerID="98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2" exitCode=0
Nov 03 19:52:13.023672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.023545    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"
Nov 03 19:52:13.023672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.023578    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" event={"ID":"a9010198-cb29-4513-a776-db4a17c8c0ed","Type":"ContainerDied","Data":"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"}
Nov 03 19:52:13.023672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.023627    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj" event={"ID":"a9010198-cb29-4513-a776-db4a17c8c0ed","Type":"ContainerDied","Data":"ce9300a8aa1dd0b8ac21fa04a7ca8b722215c206c0ac8dd939af7db1c1590f6c"}
Nov 03 19:52:13.023672 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.023646    4944 scope.go:122] "RemoveContainer" containerID="98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"
Nov 03 19:52:13.032530 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.032510    4944 scope.go:122] "RemoveContainer" containerID="98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"
Nov 03 19:52:13.032779 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:52:13.032756    4944 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2\": container with ID starting with 98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2 not found: ID does not exist" containerID="98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"
Nov 03 19:52:13.032875 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.032811    4944 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2"} err="failed to get container status \"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2\": rpc error: code = NotFound desc = could not find container \"98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2\": container with ID starting with 98e9374201b1d6fe94ad739235e7644947332fe95446e4b24d392e296a57e2f2 not found: ID does not exist"
Nov 03 19:52:13.178667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.177898    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:52:13.182662 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.182625    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8840718e-kserve-56f5579fd5-tczgj"]
Nov 03 19:52:13.932738 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:13.932691    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" path="/var/lib/kubelet/pods/a9010198-cb29-4513-a776-db4a17c8c0ed/volumes"
Nov 03 19:52:35.224158 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:52:35.224122    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:52:35.282365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282338    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"]
Nov 03 19:52:35.282743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282730    4944 cpu_manager.go:414] "containerMap: removing container" podUID="0f9cd939-4d90-4281-9868-65004ae59e32" containerName="revoke-keys"
Nov 03 19:52:35.282804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282744    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9cd939-4d90-4281-9868-65004ae59e32" containerName="revoke-keys"
Nov 03 19:52:35.282804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282758    4944 cpu_manager.go:414] "containerMap: removing container" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerName="main"
Nov 03 19:52:35.282804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282763    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerName="main"
Nov 03 19:52:35.282911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282857    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9cd939-4d90-4281-9868-65004ae59e32" containerName="revoke-keys"
Nov 03 19:52:35.282911 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.282869    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9010198-cb29-4513-a776-db4a17c8c0ed" containerName="main"
Nov 03 19:52:35.287658 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.287632    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.325365 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.325335    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.325514 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.325394    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.325566 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.325509    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prh97\" (UniqueName: \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.336458 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.336425    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:52:35.336593 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.336440    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:52:35.382493 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.382457    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:52:35.382654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.382496    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:52:35.384650 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.384628    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"]
Nov 03 19:52:35.426257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.426234    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.426414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.426285    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.426414 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.426331    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prh97\" (UniqueName: \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.426959 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.426938    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.428654 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.428631    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.484354 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.484292    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prh97\" (UniqueName: \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\") pod \"maas-api-revoke-sub-3c448465-hd2vb\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.601171 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.601146    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:35.968367 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:35.968332    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"]
Nov 03 19:52:35.975202 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:52:35.975173    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a669c5_d82d_4823_95f2_5f32f605b1e5.slice/crio-7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702 WatchSource:0}: Error finding container 7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702: Status 404 returned error can't find the container with id 7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702
Nov 03 19:52:36.171518 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:36.171484    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb" event={"ID":"10a669c5-d82d-4823-95f2-5f32f605b1e5","Type":"ContainerStarted","Data":"7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702"}
Nov 03 19:52:37.181079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:37.180986    4944 generic.go:356] "Generic (PLEG): container finished" podID="10a669c5-d82d-4823-95f2-5f32f605b1e5" containerID="66bb86a6abe6f07f72df6ec8db4fb540de812387dd429240c86885732324b20e" exitCode=0
Nov 03 19:52:37.181079 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:37.181060    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb" event={"ID":"10a669c5-d82d-4823-95f2-5f32f605b1e5","Type":"ContainerDied","Data":"66bb86a6abe6f07f72df6ec8db4fb540de812387dd429240c86885732324b20e"}
Nov 03 19:52:38.317495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.317466    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:38.453749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.453665    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\") pod \"10a669c5-d82d-4823-95f2-5f32f605b1e5\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") "
Nov 03 19:52:38.453749 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.453715    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\" (UniqueName: \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\") pod \"10a669c5-d82d-4823-95f2-5f32f605b1e5\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") "
Nov 03 19:52:38.454042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.453803    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\") pod \"10a669c5-d82d-4823-95f2-5f32f605b1e5\" (UID: \"10a669c5-d82d-4823-95f2-5f32f605b1e5\") "
Nov 03 19:52:38.454116 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.454090    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca" pod "10a669c5-d82d-4823-95f2-5f32f605b1e5" (UID: "10a669c5-d82d-4823-95f2-5f32f605b1e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:52:38.454221 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.454199    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.455942 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.455913    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca" pod "10a669c5-d82d-4823-95f2-5f32f605b1e5" (UID: "10a669c5-d82d-4823-95f2-5f32f605b1e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:52:38.456052 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.455984    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97" pod "10a669c5-d82d-4823-95f2-5f32f605b1e5" (UID: "10a669c5-d82d-4823-95f2-5f32f605b1e5"). InnerVolumeSpecName "kube-api-access-prh97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:52:38.555486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.555441    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prh97\" (UniqueName: \"kubernetes.io/projected/10a669c5-d82d-4823-95f2-5f32f605b1e5-kube-api-access-prh97\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:38.555486 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:38.555481    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a669c5-d82d-4823-95f2-5f32f605b1e5-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:52:39.192165 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:39.192132    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb"
Nov 03 19:52:39.192340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:39.192127    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c448465-hd2vb" event={"ID":"10a669c5-d82d-4823-95f2-5f32f605b1e5","Type":"ContainerDied","Data":"7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702"}
Nov 03 19:52:39.192340 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:39.192265    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ce0b730c15d54c27fe5d649da90c8ed6436b758d14c1e95186bb9ca72d12702"
Nov 03 19:52:40.325216 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:40.325178    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:40.574769 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:40.574735    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:52:58.045112 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:58.045062    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"]
Nov 03 19:52:58.090764 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:58.090720    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-s6745"]
Nov 03 19:52:59.935676 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:52:59.935631    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b90d1e-c262-4a44-b0fa-951ca63a70a0" path="/var/lib/kubelet/pods/d6b90d1e-c262-4a44-b0fa-951ca63a70a0/volumes"
Nov 03 19:53:05.223049 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:53:05.223014    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:53:18.562075 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.561991    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"]
Nov 03 19:53:18.562544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.562433    4944 cpu_manager.go:414] "containerMap: removing container" podUID="10a669c5-d82d-4823-95f2-5f32f605b1e5" containerName="revoke-keys"
Nov 03 19:53:18.562544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.562449    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a669c5-d82d-4823-95f2-5f32f605b1e5" containerName="revoke-keys"
Nov 03 19:53:18.562544 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.562538    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a669c5-d82d-4823-95f2-5f32f605b1e5" containerName="revoke-keys"
Nov 03 19:53:18.565940 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.565909    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.614623 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.614600    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:53:18.642833 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.642804    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.642965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.642876    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.642965 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.642952    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.660602 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.660577    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:53:18.660743 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.660698    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:53:18.660824 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.660749    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:53:18.662620 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.662599    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"]
Nov 03 19:53:18.743513 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.743489    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.743659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.743525    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.743659 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.743572    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.744252 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.744224    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.745919 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.745892    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.800988 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.800957    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\") pod \"maas-api-revoke-sub-f03629db-vw5tm\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:18.878928 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:18.878838    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:19.245516 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:19.245485    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"]
Nov 03 19:53:19.252453 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:53:19.252421    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa63237_fc7b_4ed1_a1d1_c5738f137f1a.slice/crio-74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041 WatchSource:0}: Error finding container 74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041: Status 404 returned error can't find the container with id 74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041
Nov 03 19:53:19.396305 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:19.396274    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm" event={"ID":"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a","Type":"ContainerStarted","Data":"74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041"}
Nov 03 19:53:20.048830 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:20.048781    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"]
Nov 03 19:53:20.094865 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:20.094836    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-qwvxg"]
Nov 03 19:53:20.403577 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:20.403497    4944 generic.go:356] "Generic (PLEG): container finished" podID="5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" containerID="530fa1cd76ae4d9211c394126e64fd5c16b2509ac77ed3c1bf0e8bb38274e215" exitCode=0
Nov 03 19:53:20.403760 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:20.403589    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm" event={"ID":"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a","Type":"ContainerDied","Data":"530fa1cd76ae4d9211c394126e64fd5c16b2509ac77ed3c1bf0e8bb38274e215"}
Nov 03 19:53:21.540276 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.540252    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:21.569508 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.569483    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\") pod \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") "
Nov 03 19:53:21.569779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.569515    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\") pod \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") "
Nov 03 19:53:21.569779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.569632    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\") pod \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\" (UID: \"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a\") "
Nov 03 19:53:21.569966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.569815    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca" pod "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" (UID: "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:21.569966 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.569915    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:21.571610 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.571590    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5" pod "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" (UID: "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a"). InnerVolumeSpecName "kube-api-access-dmrq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:21.571697 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.571679    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca" pod "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" (UID: "5aa63237-fc7b-4ed1-a1d1-c5738f137f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:21.670560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.670486    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmrq5\" (UniqueName: \"kubernetes.io/projected/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-kube-api-access-dmrq5\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:21.670560 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.670514    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5aa63237-fc7b-4ed1-a1d1-c5738f137f1a-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:21.939334 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:21.939257    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93fcf900-e509-483c-a534-b9a45e46f5d1" path="/var/lib/kubelet/pods/93fcf900-e509-483c-a534-b9a45e46f5d1/volumes"
Nov 03 19:53:22.415600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:22.415570    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm"
Nov 03 19:53:22.415600 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:22.415584    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vw5tm" event={"ID":"5aa63237-fc7b-4ed1-a1d1-c5738f137f1a","Type":"ContainerDied","Data":"74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041"}
Nov 03 19:53:22.415932 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:22.415628    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74c476901ecdb5d61c976875a6b65b5cf006a99bc3cd4cfb2ffbe1f25f746041"
Nov 03 19:53:35.289353 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:53:35.289320    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:53:40.449440 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.449405    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"]
Nov 03 19:53:40.449883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.449784    4944 cpu_manager.go:414] "containerMap: removing container" podUID="5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" containerName="revoke-keys"
Nov 03 19:53:40.449883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.449810    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" containerName="revoke-keys"
Nov 03 19:53:40.449974 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.449898    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa63237-fc7b-4ed1-a1d1-c5738f137f1a" containerName="revoke-keys"
Nov 03 19:53:40.453027 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.452998    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.501838 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.501805    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:53:40.541159 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.541130    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.541322 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.541254    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.541370 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.541319    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgh4\" (UniqueName: \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.548039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.548006    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:53:40.548039 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.548015    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:53:40.548239 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.548042    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:53:40.549925 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.549902    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"]
Nov 03 19:53:40.642338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.642307    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.642504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.642363    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgh4\" (UniqueName: \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.642504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.642390    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.643011 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.642989    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.644699 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.644675    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.677978 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.677916    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:53:40.699373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.699342    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgh4\" (UniqueName: \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\") pod \"maas-api-revoke-sub-0165d7d2-z65k4\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:40.766930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:40.766907    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:41.134826 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:41.134717    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"]
Nov 03 19:53:41.142220 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:53:41.142193    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb2ef74e_d8c1_4304_b169_3d9f43608907.slice/crio-7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf WatchSource:0}: Error finding container 7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf: Status 404 returned error can't find the container with id 7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf
Nov 03 19:53:41.524689 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:41.524646    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4" event={"ID":"fb2ef74e-d8c1-4304-b169-3d9f43608907","Type":"ContainerStarted","Data":"02356dd6de6f26b216190cadd0662cd7bc7a6101960a2ea3808ce2c231b1df7b"}
Nov 03 19:53:41.525097 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:41.524710    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4" event={"ID":"fb2ef74e-d8c1-4304-b169-3d9f43608907","Type":"ContainerStarted","Data":"7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf"}
Nov 03 19:53:41.633295 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:41.633243    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4" podStartSLOduration=1.6332296739999999 podStartE2EDuration="1.633229674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:53:41.632348029 +0000 UTC m=+4119.909701612" watchObservedRunningTime="2026-11-03 19:53:41.633229674 +0000 UTC m=+4119.910583256"
Nov 03 19:53:42.530358 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:42.530325    4944 generic.go:356] "Generic (PLEG): container finished" podID="fb2ef74e-d8c1-4304-b169-3d9f43608907" containerID="02356dd6de6f26b216190cadd0662cd7bc7a6101960a2ea3808ce2c231b1df7b" exitCode=0
Nov 03 19:53:42.530772 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:42.530409    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4" event={"ID":"fb2ef74e-d8c1-4304-b169-3d9f43608907","Type":"ContainerDied","Data":"02356dd6de6f26b216190cadd0662cd7bc7a6101960a2ea3808ce2c231b1df7b"}
Nov 03 19:53:43.664071 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.664041    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:43.771478 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.771448    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\" (UniqueName: \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\") pod \"fb2ef74e-d8c1-4304-b169-3d9f43608907\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") "
Nov 03 19:53:43.771657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.771590    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\") pod \"fb2ef74e-d8c1-4304-b169-3d9f43608907\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") "
Nov 03 19:53:43.771657 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.771650    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\") pod \"fb2ef74e-d8c1-4304-b169-3d9f43608907\" (UID: \"fb2ef74e-d8c1-4304-b169-3d9f43608907\") "
Nov 03 19:53:43.772024 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.771997    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca" pod "fb2ef74e-d8c1-4304-b169-3d9f43608907" (UID: "fb2ef74e-d8c1-4304-b169-3d9f43608907"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:43.773494 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.773468    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4" pod "fb2ef74e-d8c1-4304-b169-3d9f43608907" (UID: "fb2ef74e-d8c1-4304-b169-3d9f43608907"). InnerVolumeSpecName "kube-api-access-sxgh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:43.773583 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.773567    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca" pod "fb2ef74e-d8c1-4304-b169-3d9f43608907" (UID: "fb2ef74e-d8c1-4304-b169-3d9f43608907"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:43.873093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.873013    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgh4\" (UniqueName: \"kubernetes.io/projected/fb2ef74e-d8c1-4304-b169-3d9f43608907-kube-api-access-sxgh4\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:43.873093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.873044    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:43.873093 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:43.873055    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb2ef74e-d8c1-4304-b169-3d9f43608907-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:44.540625 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:44.540593    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4"
Nov 03 19:53:44.540841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:44.540595    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z65k4" event={"ID":"fb2ef74e-d8c1-4304-b169-3d9f43608907","Type":"ContainerDied","Data":"7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf"}
Nov 03 19:53:44.540841 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:44.540728    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7352311bd426b3f9844fa0f62cd55fad2b80a6487de7bba0833f0d5aae44d1cf"
Nov 03 19:53:51.862051 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.862014    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"]
Nov 03 19:53:51.862459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.862436    4944 cpu_manager.go:414] "containerMap: removing container" podUID="fb2ef74e-d8c1-4304-b169-3d9f43608907" containerName="revoke-keys"
Nov 03 19:53:51.862459 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.862448    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb2ef74e-d8c1-4304-b169-3d9f43608907" containerName="revoke-keys"
Nov 03 19:53:51.862535 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.862520    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb2ef74e-d8c1-4304-b169-3d9f43608907" containerName="revoke-keys"
Nov 03 19:53:51.866028 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.866006    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:51.914993 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.914959    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 19:53:51.946908 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.946876    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:51.947083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.946977    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:51.947083 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.947067    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zrz\" (UniqueName: \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:51.961164 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.961139    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wqkjw\""
Nov 03 19:53:51.961341 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.961325    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 19:53:51.961426 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.961388    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 19:53:51.963189 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:51.963170    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"]
Nov 03 19:53:52.048377 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.048340    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.048564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.048402    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.048564 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.048436    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zrz\" (UniqueName: \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.049056 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.049031    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.050726 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.050700    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.105594 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.105561    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zrz\" (UniqueName: \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\") pod \"maas-api-revoke-sub-6b19338f-n495m\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.178483 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.178405    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:52.545248 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.545213    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"]
Nov 03 19:53:52.551493 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:53:52.551468    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a4eaf1_fa8a_4908_9e70_2d5919d63580.slice/crio-dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311 WatchSource:0}: Error finding container dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311: Status 404 returned error can't find the container with id dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311
Nov 03 19:53:52.586831 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:52.586783    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m" event={"ID":"43a4eaf1-fa8a-4908-9e70-2d5919d63580","Type":"ContainerStarted","Data":"dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311"}
Nov 03 19:53:53.593823 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:53.593773    4944 generic.go:356] "Generic (PLEG): container finished" podID="43a4eaf1-fa8a-4908-9e70-2d5919d63580" containerID="c5e547d3f5fdd554bc68194f1ea18ffd478bcf6a41f31d3a316ccf4ce5dd59a6" exitCode=0
Nov 03 19:53:53.594214 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:53.593834    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m" event={"ID":"43a4eaf1-fa8a-4908-9e70-2d5919d63580","Type":"ContainerDied","Data":"c5e547d3f5fdd554bc68194f1ea18ffd478bcf6a41f31d3a316ccf4ce5dd59a6"}
Nov 03 19:53:54.728006 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.727979    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:54.875515 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.875439    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\") pod \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") "
Nov 03 19:53:54.875667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.875565    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\") pod \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") "
Nov 03 19:53:54.875667 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.875592    4944 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\" (UniqueName: \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\") pod \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\" (UID: \"43a4eaf1-fa8a-4908-9e70-2d5919d63580\") "
Nov 03 19:53:54.875802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.875757    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca" pod "43a4eaf1-fa8a-4908-9e70-2d5919d63580" (UID: "43a4eaf1-fa8a-4908-9e70-2d5919d63580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 19:53:54.877736 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.877713    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca" pod "43a4eaf1-fa8a-4908-9e70-2d5919d63580" (UID: "43a4eaf1-fa8a-4908-9e70-2d5919d63580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 19:53:54.877868 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.877777    4944 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz" pod "43a4eaf1-fa8a-4908-9e70-2d5919d63580" (UID: "43a4eaf1-fa8a-4908-9e70-2d5919d63580"). InnerVolumeSpecName "kube-api-access-k4zrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 19:53:54.976946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.976917    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-xks-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:54.976946 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.976941    4944 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4zrz\" (UniqueName: \"kubernetes.io/projected/43a4eaf1-fa8a-4908-9e70-2d5919d63580-kube-api-access-k4zrz\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:54.977126 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:54.976952    4944 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43a4eaf1-fa8a-4908-9e70-2d5919d63580-maas-api-openshift-ca\") on node \"ip-10-0-137-192.us-west-2.compute.internal\" DevicePath \"\""
Nov 03 19:53:55.612321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:55.612275    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m" event={"ID":"43a4eaf1-fa8a-4908-9e70-2d5919d63580","Type":"ContainerDied","Data":"dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311"}
Nov 03 19:53:55.612321 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:55.612311    4944 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-n495m"
Nov 03 19:53:55.612646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:53:55.612443    4944 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfa4b2e38794d3b6de22c0733702c4fc619dacd38acc94a9b4e28c5f2f5b2311"
Nov 03 19:54:04.217162 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:04.217125    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf8d44fc4-w4tl2_2c3a53ea-a998-4524-b09e-c0ef5faec585/authorino/0.log"
Nov 03 19:54:05.291798 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:54:05.291749    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:54:06.229933 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:06.229896    4944 scope.go:122] "RemoveContainer" containerID="c25dc3a3406fabad1740d61e0ffe97dd29ae6aa5a3f3676c477bec64775db390"
Nov 03 19:54:16.624517 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:16.624484    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tx2vn_144204e6-4a23-4dfe-9686-e6da6c450145/server/0.log"
Nov 03 19:54:16.854702 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:16.854674    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-h2qzd_3d3b180a-7891-49c6-9656-1eb34b3da879/manager/0.log"
Nov 03 19:54:17.366682 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:17.366622    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-fcklg_0f9f5095-7613-41d5-b161-a58fac039e4a/manager/0.log"
Nov 03 19:54:17.434143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:17.434110    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-fcklg_0f9f5095-7613-41d5-b161-a58fac039e4a/copy-manifests/0.log"
Nov 03 19:54:19.547711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:19.547683    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z65k4_fb2ef74e-d8c1-4304-b169-3d9f43608907/revoke-keys/0.log"
Nov 03 19:54:19.768337 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:19.768303    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c448465-hd2vb_10a669c5-d82d-4823-95f2-5f32f605b1e5/revoke-keys/0.log"
Nov 03 19:54:19.983317 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:19.983242    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qlnk9_0f9cd939-4d90-4281-9868-65004ae59e32/revoke-keys/0.log"
Nov 03 19:54:20.209060 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:20.209032    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-zv87c_b3538d60-e0e8-4a71-9ced-0197bc7e45a7/revoke-keys/0.log"
Nov 03 19:54:20.437863 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:20.437828    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-469481af-682df_335d34c9-8094-4ef8-975c-d12966b2e670/revoke-keys/0.log"
Nov 03 19:54:20.657930 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:20.657901    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n495m_43a4eaf1-fa8a-4908-9e70-2d5919d63580/revoke-keys/0.log"
Nov 03 19:54:20.868992 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:20.868961    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vw5tm_5aa63237-fc7b-4ed1-a1d1-c5738f137f1a/revoke-keys/0.log"
Nov 03 19:54:22.610562 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:22.610510    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf8d44fc4-w4tl2_2c3a53ea-a998-4524-b09e-c0ef5faec585/authorino/0.log"
Nov 03 19:54:23.072542 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:23.072513    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-ws6kr_5b34af26-bee6-4b33-983a-29ef3f37657a/manager/0.log"
Nov 03 19:54:25.798802 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:25.798772    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc56b666-vbh86_65da9844-a098-4c47-8c70-6e1b8b923b25/kube-auth-proxy/0.log"
Nov 03 19:54:27.403143 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:27.403108    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-gwxrp_55c38327-6415-43dc-b53b-8d2033f1bd4e/istio-proxy/0.log"
Nov 03 19:54:27.630259 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:27.630229    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h6d6n_c94260b7-d303-41a2-b800-ce732a0b3a8b/istio-proxy/0.log"
Nov 03 19:54:27.846955 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:27.846922    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-stw9w_19b4623f-6a4c-4ed0-bad9-7883737d362a/istio-proxy/0.log"
Nov 03 19:54:28.482347 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:28.482316    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:54:29.515235 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:29.515205    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr_0b0bd44e-1b16-4195-9d46-c3a59e9a5534/main/0.log"
Nov 03 19:54:30.196952 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:30.196922    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8nk8p_dd152083-a780-4dcf-8d62-884d530b281d/main/0.log"
Nov 03 19:54:30.661585 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:30.661555    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn_3ef9f491-f8de-46a5-b4c4-de5981ca5ff2/main/0.log"
Nov 03 19:54:30.868581 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:30.868553    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7_e1176c49-737c-4882-905e-799b99aa88fb/main/0.log"
Nov 03 19:54:31.747622 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:31.747587    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf8d44fc4-w4tl2_2c3a53ea-a998-4524-b09e-c0ef5faec585/authorino/0.log"
Nov 03 19:54:35.225099 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:54:35.225060    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:54:37.053829 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:37.053776    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"]
Nov 03 19:54:37.099843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:37.099814    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-469481af-682df"]
Nov 03 19:54:37.935335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:37.935301    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="335d34c9-8094-4ef8-975c-d12966b2e670" path="/var/lib/kubelet/pods/335d34c9-8094-4ef8-975c-d12966b2e670/volumes"
Nov 03 19:54:41.028752 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:41.028669    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:54:42.248300 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:42.248269    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-tx2vn_144204e6-4a23-4dfe-9686-e6da6c450145/server/0.log"
Nov 03 19:54:42.479569 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:42.479534    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-h2qzd_3d3b180a-7891-49c6-9656-1eb34b3da879/manager/0.log"
Nov 03 19:54:42.993107 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:42.993079    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-fcklg_0f9f5095-7613-41d5-b161-a58fac039e4a/copy-manifests/0.log"
Nov 03 19:54:43.059390 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:43.059358    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5ff9d8584c-fcklg_0f9f5095-7613-41d5-b161-a58fac039e4a/manager/0.log"
Nov 03 19:54:44.901843 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:44.901814    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z65k4_fb2ef74e-d8c1-4304-b169-3d9f43608907/revoke-keys/0.log"
Nov 03 19:54:45.126742 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:45.126712    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3c448465-hd2vb_10a669c5-d82d-4823-95f2-5f32f605b1e5/revoke-keys/0.log"
Nov 03 19:54:45.353558 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:45.353522    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-qlnk9_0f9cd939-4d90-4281-9868-65004ae59e32/revoke-keys/0.log"
Nov 03 19:54:45.572135 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:45.572108    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-zv87c_b3538d60-e0e8-4a71-9ced-0197bc7e45a7/revoke-keys/0.log"
Nov 03 19:54:45.788091 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:45.788063    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-n495m_43a4eaf1-fa8a-4908-9e70-2d5919d63580/revoke-keys/0.log"
Nov 03 19:54:46.011814 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:46.011766    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vw5tm_5aa63237-fc7b-4ed1-a1d1-c5738f137f1a/revoke-keys/0.log"
Nov 03 19:54:47.345338 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:47.345303    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf8d44fc4-w4tl2_2c3a53ea-a998-4524-b09e-c0ef5faec585/authorino/0.log"
Nov 03 19:54:47.729057 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:47.728974    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-ws6kr_5b34af26-bee6-4b33-983a-29ef3f37657a/manager/0.log"
Nov 03 19:54:50.240907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:50.240878    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-86dc56b666-vbh86_65da9844-a098-4c47-8c70-6e1b8b923b25/kube-auth-proxy/0.log"
Nov 03 19:54:51.822308 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:51.822281    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-gwxrp_55c38327-6415-43dc-b53b-8d2033f1bd4e/istio-proxy/0.log"
Nov 03 19:54:52.058160 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:52.058135    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-h6d6n_c94260b7-d303-41a2-b800-ce732a0b3a8b/istio-proxy/0.log"
Nov 03 19:54:52.285589 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:52.285563    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-stw9w_19b4623f-6a4c-4ed0-bad9-7883737d362a/istio-proxy/0.log"
Nov 03 19:54:52.971231 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:52.971200    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:54:53.827373 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:53.827343    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-vp8wr_0b0bd44e-1b16-4195-9d46-c3a59e9a5534/main/0.log"
Nov 03 19:54:54.523846 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:54.523812    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-8nk8p_dd152083-a780-4dcf-8d62-884d530b281d/main/0.log"
Nov 03 19:54:54.976779 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:54.976700    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-6xkxn_3ef9f491-f8de-46a5-b4c4-de5981ca5ff2/main/0.log"
Nov 03 19:54:55.214533 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:54:55.214502    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wdkl7_e1176c49-737c-4882-905e-799b99aa88fb/main/0.log"
Nov 03 19:55:02.571105 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:02.571075    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:55:02.572584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:02.572562    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5g7pb_8a9aec1c-d239-4cd6-b8f6-c6000e4cb6b0/ovn-acl-logging/0.log"
Nov 03 19:55:05.224076 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:55:05.224042    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:55:06.290711 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:06.290679    4944 scope.go:122] "RemoveContainer" containerID="2bc483d310414907d1d52e17a789e20c056d56c738e6626a18c9a10c373789e2"
Nov 03 19:55:28.299042 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:28.299011    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:55:28.520646 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:28.520613    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:55:35.223680 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:55:35.223643    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:55:41.413055 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:55:41.413021    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:05.223095 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:56:05.223055    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:56:06.309997 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:06.309963    4944 scope.go:122] "RemoveContainer" containerID="bf2433b410e5c29c5cdb6e21c4430faa2f6794fb5e18e0506a55016ac1be55a4"
Nov 03 19:56:06.318125 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:06.318103    4944 scope.go:122] "RemoveContainer" containerID="bb6e732ead6fcf6e6ff8f090fbab3ad81752f2bb792939cb29c600c7e6b04888"
Nov 03 19:56:08.253401 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:08.253363    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:08.422599 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:08.422572    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:56:08.647152 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:08.647079    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m756l_aafa7ea1-c312-4a7b-8d9e-72cb169f0c6a/payload-processing/0.log"
Nov 03 19:56:09.016409 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.016371    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t5k7x/must-gather-4t4bf"]
Nov 03 19:56:09.016883 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.016864    4944 cpu_manager.go:414] "containerMap: removing container" podUID="43a4eaf1-fa8a-4908-9e70-2d5919d63580" containerName="revoke-keys"
Nov 03 19:56:09.017009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.016885    4944 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a4eaf1-fa8a-4908-9e70-2d5919d63580" containerName="revoke-keys"
Nov 03 19:56:09.017009 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.016988    4944 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a4eaf1-fa8a-4908-9e70-2d5919d63580" containerName="revoke-keys"
Nov 03 19:56:09.020427 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.020405    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.036191 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.036161    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a85baec3-c2c8-4c1f-9eec-4840540fcc18-must-gather-output\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.036328 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.036206    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztjs9\" (UniqueName: \"kubernetes.io/projected/a85baec3-c2c8-4c1f-9eec-4840540fcc18-kube-api-access-ztjs9\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.069090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.069065    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"openshift-service-ca.crt\""
Nov 03 19:56:09.069197 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.069152    4944 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t5k7x\"/\"kube-root-ca.crt\""
Nov 03 19:56:09.100935 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.100904    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.115139 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.115115    4944 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t5k7x\"/\"default-dockercfg-c8qsb\""
Nov 03 19:56:09.117119 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.117097    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/must-gather-4t4bf"]
Nov 03 19:56:09.137124 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.137095    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a85baec3-c2c8-4c1f-9eec-4840540fcc18-must-gather-output\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.137256 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.137149    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztjs9\" (UniqueName: \"kubernetes.io/projected/a85baec3-c2c8-4c1f-9eec-4840540fcc18-kube-api-access-ztjs9\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.137434 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.137416    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a85baec3-c2c8-4c1f-9eec-4840540fcc18-must-gather-output\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.194422 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.194393    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztjs9\" (UniqueName: \"kubernetes.io/projected/a85baec3-c2c8-4c1f-9eec-4840540fcc18-kube-api-access-ztjs9\") pod \"must-gather-4t4bf\" (UID: \"a85baec3-c2c8-4c1f-9eec-4840540fcc18\") " pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.333907 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.333835    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/must-gather-4t4bf"
Nov 03 19:56:09.353053 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.353019    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:09.699584 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.699511    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/must-gather-4t4bf"]
Nov 03 19:56:09.706857 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:56:09.706824    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda85baec3_c2c8_4c1f_9eec_4840540fcc18.slice/crio-563fddcecfbd9f5942e3ed8854a39c887abe5e0c2c06658de79ab0748ef68b43 WatchSource:0}: Error finding container 563fddcecfbd9f5942e3ed8854a39c887abe5e0c2c06658de79ab0748ef68b43: Status 404 returned error can't find the container with id 563fddcecfbd9f5942e3ed8854a39c887abe5e0c2c06658de79ab0748ef68b43
Nov 03 19:56:09.708804 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:09.708768    4944 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 19:56:10.267612 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:10.267571    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/must-gather-4t4bf" event={"ID":"a85baec3-c2c8-4c1f-9eec-4840540fcc18","Type":"ContainerStarted","Data":"563fddcecfbd9f5942e3ed8854a39c887abe5e0c2c06658de79ab0748ef68b43"}
Nov 03 19:56:11.274575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:11.274516    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/must-gather-4t4bf" event={"ID":"a85baec3-c2c8-4c1f-9eec-4840540fcc18","Type":"ContainerStarted","Data":"f9d0a5d9994a68cc324fca551b3d7e7414918d000a3231774640bb0c24df50cc"}
Nov 03 19:56:11.274575 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:11.274558    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/must-gather-4t4bf" event={"ID":"a85baec3-c2c8-4c1f-9eec-4840540fcc18","Type":"ContainerStarted","Data":"6f9b1f1d7007d279fc3e12b522f86901a77b0ab49ca9a559c418da8db36dc8da"}
Nov 03 19:56:11.383405 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:11.383355    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t5k7x/must-gather-4t4bf" podStartSLOduration=2.076879292 podStartE2EDuration="3.383341972s" totalImagesPullingTime="1.30646268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:11.381325114 +0000 UTC m=+4269.658678697" watchObservedRunningTime="2026-11-03 19:56:11.383341972 +0000 UTC m=+4269.660695571"
Nov 03 19:56:17.459429 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:17.459390    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-94hv8_bca87fa3-f076-4255-9cb1-6a93dac8ff63/global-pull-secret-syncer/0.log"
Nov 03 19:56:20.499580 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:20.499527    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log"
Nov 03 19:56:22.116062 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:22.116025    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-192.us-west-2.compute.internal_5b200ea252cb76e83bc321c9ddace574/haproxy/0.log"
Nov 03 19:56:27.220335 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:27.220295    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6cf8d44fc4-w4tl2_2c3a53ea-a998-4524-b09e-c0ef5faec585/authorino/0.log"
Nov 03 19:56:27.542953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:27.542924    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-ws6kr_5b34af26-bee6-4b33-983a-29ef3f37657a/manager/0.log"
Nov 03 19:56:27.997733 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:27.997691    4944 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"]
Nov 03 19:56:28.004332 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.004304    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.059144 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.059103    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"]
Nov 03 19:56:28.147446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.147407    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-sys\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.147446 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.147450    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-podres\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.147698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.147554    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-proc\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.147698 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.147668    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-lib-modules\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.147777 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.147717    4944 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cqm\" (UniqueName: \"kubernetes.io/projected/12e41c75-af82-4f5c-84ee-d32ca346c3b9-kube-api-access-w2cqm\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.248588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248492    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-lib-modules\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.248588 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248578    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cqm\" (UniqueName: \"kubernetes.io/projected/12e41c75-af82-4f5c-84ee-d32ca346c3b9-kube-api-access-w2cqm\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248627    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-lib-modules\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248637    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-sys\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248697    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-sys\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248725    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-podres\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248781    4944 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-proc\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.248978    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-proc\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.249212 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.249045    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/12e41c75-af82-4f5c-84ee-d32ca346c3b9-podres\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.308356 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.308276    4944 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cqm\" (UniqueName: \"kubernetes.io/projected/12e41c75-af82-4f5c-84ee-d32ca346c3b9-kube-api-access-w2cqm\") pod \"perf-node-gather-daemonset-mp4qk\" (UID: \"12e41c75-af82-4f5c-84ee-d32ca346c3b9\") " pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.316759 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.316731    4944 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:28.691669 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:28.691594    4944 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"]
Nov 03 19:56:28.698324 ip-10-0-137-192 kubenswrapper[4944]: W1103 19:56:28.698292    4944 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod12e41c75_af82_4f5c_84ee_d32ca346c3b9.slice/crio-e99340404a11998c778260619fb21140eb3d1ae403a74780cfad50ff8cd0f05e WatchSource:0}: Error finding container e99340404a11998c778260619fb21140eb3d1ae403a74780cfad50ff8cd0f05e: Status 404 returned error can't find the container with id e99340404a11998c778260619fb21140eb3d1ae403a74780cfad50ff8cd0f05e
Nov 03 19:56:29.443495 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:29.443455    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk" event={"ID":"12e41c75-af82-4f5c-84ee-d32ca346c3b9","Type":"ContainerStarted","Data":"1bbb53e44c04df484bfce9bfcf8b9573996305396a719a2b29888064cf29794b"}
Nov 03 19:56:29.443929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:29.443519    4944 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk" event={"ID":"12e41c75-af82-4f5c-84ee-d32ca346c3b9","Type":"ContainerStarted","Data":"e99340404a11998c778260619fb21140eb3d1ae403a74780cfad50ff8cd0f05e"}
Nov 03 19:56:29.443929 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:29.443562    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:29.553423 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:29.553371    4944 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk" podStartSLOduration=2.5533582470000002 podStartE2EDuration="2.553358247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:29.551815647 +0000 UTC m=+4287.829169220" watchObservedRunningTime="2026-11-03 19:56:29.553358247 +0000 UTC m=+4287.830711847"
Nov 03 19:56:33.046339 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:33.046293    4944 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"]
Nov 03 19:56:33.093383 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:33.093348    4944 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zv87c"]
Nov 03 19:56:33.932651 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:33.932601    4944 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3538d60-e0e8-4a71-9ced-0197bc7e45a7" path="/var/lib/kubelet/pods/b3538d60-e0e8-4a71-9ced-0197bc7e45a7/volumes"
Nov 03 19:56:35.224444 ip-10-0-137-192 kubenswrapper[4944]: E1103 19:56:35.224397    4944 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-l8qkk" containerName="extractor"
Nov 03 19:56:35.460058 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:35.460027    4944 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t5k7x/perf-node-gather-daemonset-mp4qk"
Nov 03 19:56:36.988425 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:36.988393    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-d45fs_c4b2235a-1853-4d53-8629-cf3bc775fb65/monitoring-plugin/0.log"
Nov 03 19:56:39.855224 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:39.855195    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd5jv_a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/node-exporter/0.log"
Nov 03 19:56:40.266090 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:40.266065    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd5jv_a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/kube-rbac-proxy/0.log"
Nov 03 19:56:40.668019 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:40.667944    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gd5jv_a592af2a-0e6b-4455-b71f-bf6c8fe5a14c/init-textfile/0.log"
Nov 03 19:56:41.073953 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:41.073923    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5q2l5_ce552fcf-36e3-435c-887d-6ceff825d956/kube-rbac-proxy-main/0.log"
Nov 03 19:56:41.487008 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:41.486927    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5q2l5_ce552fcf-36e3-435c-887d-6ceff825d956/kube-rbac-proxy-self/0.log"
Nov 03 19:56:41.715845 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:41.715780    4944 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ddzdz_89884da1-8f1b-4d40-a0ca-9d168399a458/konnectivity-agent/0.log" limit=1024
Nov 03 19:56:41.889864 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:41.889833    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5q2l5_ce552fcf-36e3-435c-887d-6ceff825d956/openshift-state-metrics/0.log"
Nov 03 19:56:42.302753 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:42.302701    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/prometheus/0.log"
Nov 03 19:56:42.705731 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:42.705653    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/config-reloader/0.log"
Nov 03 19:56:43.113747 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:43.113713    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/thanos-sidecar/0.log"
Nov 03 19:56:43.518809 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:43.517125    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/kube-rbac-proxy-web/0.log"
Nov 03 19:56:43.925756 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:43.925690    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/kube-rbac-proxy/0.log"
Nov 03 19:56:44.331619 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:44.331584    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/kube-rbac-proxy-thanos/0.log"
Nov 03 19:56:44.739820 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:44.739778    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f82db8a5-a394-4bf9-b9a8-27cd5e6fd80a/init-config-reloader/0.log"
Nov 03 19:56:47.580204 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:47.580170    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/thanos-query/0.log"
Nov 03 19:56:47.985320 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:47.985244    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/kube-rbac-proxy-web/0.log"
Nov 03 19:56:48.388257 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:48.388228    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/kube-rbac-proxy/0.log"
Nov 03 19:56:48.796070 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:48.796043    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/prom-label-proxy/0.log"
Nov 03 19:56:49.199172 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:49.199087    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/kube-rbac-proxy-rules/0.log"
Nov 03 19:56:49.601504 ip-10-0-137-192 kubenswrapper[4944]: I1103 19:56:49.601474    4944 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74994bbc58-d8mv2_618c170c-3333-4548-864b-2e1d78308333/kube-rbac-proxy-metrics/0.log"